# Optimal Transport for Offline Imitation Learning

Yicheng Luo
University College London &Zhengyao Jiang
University College London &Samuel Cohen
University College London Edward Grefenstette
University College London & Cohere &Marc Peter Deisenroth
University College London

###### Abstract

With the advent of large datasets, offline reinforcement learning (RL) is a
promising framework for learning good decision-making policies without the
need to interact with the real environment. However, offline RL requires the
dataset to be reward-annotated, which presents practical challenges when
reward engineering is difficult or when obtaining reward annotations is labor-
intensive. In this paper, we introduce Optimal Transport Reward labeling
(OTR), an algorithm that assigns rewards to offline trajectories, with a few
high-quality demonstrations. OTR’s key idea is to use optimal transport to
compute an optimal alignment between an unlabeled trajectory in the dataset
and an expert demonstration to obtain a similarity measure that can be
interpreted as a reward, which can then be used by an offline RL algorithm to
learn the policy. OTR is easy to implement and computationally efficient. On
D4RL benchmarks, we show that OTR with a single demonstration can consistently
match the performance of offline RL with ground-truth rewards111Code is
available at https://github.com/ethanluoyc/optimal_transport_reward.

##  1 Introduction

Offline Reinforcement Learning (RL) has made significant progress recently,
enabling learning policies from logged experience without any interaction with
the environment. Offline RL is relevant when online data collection can be
expensive or slow, e.g., robotics, financial trading and autonomous driving. A
key feature of offline RL is that it can learn an improved policy that goes
beyond the behavior policy that generated the data. However, offline RL
requires the existence of a reward function for labeling the logged
experience, making direct applications of offline RL methods impractical for
applications where rewards are hard to specify with hand-crafted rules. Even
if it is possible to label the trajectories with human preferences, such a
procedure to generate reward signals can be expensive. Therefore, enabling
offline RL to leverage unlabeled data is an open question of significant
practical value.

Besides labeling every single trajectory, an alternative way to inform the
agent about human preference is to provide expert demonstrations. For many
applications, providing expert demonstrations is more natural for
practitioners compared to specifying a reward function. In robotics, providing
expert demonstrations is fairly common, and in the absence of natural reward
functions, ‘learning from demonstration’ has been used for decades to find
good policies for robotic systems; see, e.g., (Atkeson & Schaal, 1997; Abbeel
& Ng, 2004; Calinon et al., 2007; Englert et al., 2013). One such framework
for learning policies from demonstrations is imitation learning (IL). IL aims
at learning policies that imitate the behavior of expert demonstrations
without an explicit reward function.

There are two popular approaches to IL: Behavior Cloning (BC) (Pomerleau,
1988) and Inverse Reinforcement Learning (IRL) (Ng & Russell, 2000). BC aims
to recover the demonstrator’s behavior directly by setting up an offline
supervised learning problem. If demonstrations are of high quality and actions
of the demonstrations are recorded, BC can work very well as demonstrated by
Pomerleau (1988), but generalization to new situations typically does not work
well. IRL learns an intermediate reward function that aims to capture the
demonstrator’s intent. Current algorithms have demonstrated very strong
empirical performance, requiring only a few expert demonstrations to obtain
good performance (Ho & Ermon, 2016; Dadashi et al., 2022). While these IRL
methods do not require many demonstrations, they typically focus on the
_online_ RL setting and require a large number of environment interactions to
learn an imitating policy, i.e., these methods are not suitable for offline
learning.

In this paper, we introduce Optimal Transport Reward labeling (OTR), an
algorithm that uses optimal transport theory to automatically assign reward
labels to unlabeled trajectories in an offline dataset, given one or more
expert demonstrations. This reward-annotated dataset can then be used by
offline RL algorithms to find good policies that imitate demonstrated
behavior. Specifically, OTR uses optimal transport to find optimal alignments
between unlabeled trajectories in the dataset and expert demonstrations. The
similarity measure between a state in unlabeled trajectory and that of an
expert trajectory is then treated as a reward label. These rewards can be used
by any offline RL algorithm for learning policies from a small number of
expert demonstrations and a large offline dataset. Figure 1 illustrates how
OTR uses expert demonstrations to add reward labels to an offline dataset,
which can then be used by an offline RL algorithm to find a good policy that
imitates demonstrated behavior. Empirical evaluations on the D4RL (Fu et al.,
2021) datasets demonstrate that OTR recovers the performance of offline RL
methods with ground-truth rewards with only a single demonstration. Compared
to previous reward learning and imitation learning approaches, our approach
also achieves consistently better performance across a wide range of offline
datasets and tasks.

Expert Demonstration$s_{1}^{e}$ $a_{2}^{e}$ $s_{2}^{e}$ $a_{2}^{e}$ $\cdots$
$s_{T}^{e}$ OTROffline Dataset$s_{1}$ $a_{2}$ $s_{2}$ $a_{2}$ $\cdots$ $s_{T}$
$r_{1}$ $r_{2}$ $\cdots$ $r_{T}$ Offline RL……

Figure 1: Illustration of Optimal Transport Reward Labeling (OTR). Given
expert demonstrations (left) and an offline dataset without reward labels
(center), OTR adds reward labels $r_{i}$ to the offline dataset by means of
optimal transport (orange, center). The labeled dataset can then be used by an
offline RL algorithm (right) to learn policies.

##  2 Offline Reinforcement Learning and Imitation Learning

##### Offline Reinforcement Learning

In offline/batch reinforcement learning, we are interested in learning
policies directly from fixed offline datasets (Lange et al., 2012; Levine et
al., 2020), i.e., the agent is not permitted any additional interaction with
the environment. Offline RL research typically assumes access to an offline
dataset of observed transitions
$\mathcal{D}=\\{(s^{i}_{t},a^{i}_{t},r^{i}_{t},s^{i}_{t+1})\\}_{i=1}^{N}$.
This setting is particularly attractive for applications where there is
previous logged experience available but online data collection is expensive
(e.g., robotics, healthcare). Recently, the field of offline RL has made
significant progress, and many offline RL algorithms have been proposed to
learn improved policies from diverse and sub-optimal offline demonstrations
(Levine et al., 2020; Fujimoto & Gu, 2021; Kumar et al., 2020; Kostrikov et
al., 2022c; Wang et al., 2020).

Offline RL research typically assumes that the offline dataset is reward-
annotated. That is, each transition
$(s^{i}_{t},a^{i}_{t},r^{i}_{t},s^{i}_{t+1})$ in the dataset is labeled with
reward $r^{i}_{t}$. One common approach to offline RL is to leverage the
reward signals in the offline dataset and learn a policy with an actor-critic
algorithm purely from offline transitions (Levine et al., 2020). However, in
practice, having per-transition reward annotations for the offline dataset may
be difficult due to the challenges in designing a good reward function. Zolna
et al. (2020) propose ORIL which learns a reward function based on positive-
unlabeled (PU) learning (Elkan & Noto, 2008) that can be used to add reward
labels to offline datasets, allowing for unlabeled datasets to be used by
offline RL algorithms.

##### Imitation Learning

Imitation Learning (IL) aims at learning policies that can mimic the behavior
of an expert demonstrator. Unlike the typical RL setting, a reward function is
not required. Instead, IL methods assume access to an expert that can provide
demonstrations of desired behavior.

There are two commonly used approaches to imitation learning. Behavior Cloning
(BC) (Pomerleau, 1988) learns a policy by learning a mapping from states to
actions in the trajectory demonstrated by an expert. By formulating policy
learning as a supervised learning problem, BC can suffer from classical
problems of supervised regression: generalizing the policy to unseen states
may not be successful due to overfitting; multiple expert trajectories that
follow different paths (bifurcation) will cause challenges. Inverse
Reinforcement Learning (IRL) considers first learning a reward function based
on expert demonstrations. The learned reward function can then be used to
train a policy using an RL algorithm. Previous work shows that IRL-based
methods can learn good policies with a small number of expert demonstrations.
However, popular recent IRL methods, such as GAIL (Ho & Ermon, 2016), require
a potentially large number of online samples during training, resulting in
poor sample efficiency. Moreover, algorithms, such as GAIL, follow a training
paradigm that is similar to Generative Adversarial Networks (GANs) (Goodfellow
et al., 2014), by formulating the problem of IRL as a minimax optimization
problem to learn a discriminator that implicitly minimizes an f-divergence. It
was found, however, that Adversarial imitation learning (AIL) methods such as
GAIL can be very difficult to optimize, requiring careful tuning of
hyperparameters (Orsini et al., 2021). Another popular approach is to perform
distribution matching between the policy and the expert demonstrations
(Englert et al., 2013; Kostrikov et al., 2022b).

More recently, IRL methods based on Optimal Transport (OT) have demonstrated
success as an alternative method for IRL compared to AIL approaches. Unlike
AIL approaches, OT methods minimize the Wasserstein distance between the
expert’s and the agent’s state-action distributions. Building on this
formulation, Xiao et al. (2019) propose to minimize the Wasserstein distance
via its dual formulation, which may lead to potential optimization issues.
More recently, Dadashi et al. (2022) introduce PWIL, which instead minimizes
the Wasserstein distance via its primal formulation, avoiding the potential
optimization issues in the dual formulation. Along this line of work, Cohen et
al. (2021) propose a series of improvements to the primal Wasserstein
formulation used in PWIL and demonstrate strong empirical results in terms of
both sample efficiency and asymptotic performance. Still, these approaches
require a large number of online samples to learn good policies. While
progress has been made to improve the sample efficiency of these approaches
(Kostrikov et al., 2022a), imitation learning without any online interaction
remains an active research area. In this paper, we provide a constructive
algorithm to address the issue of offline imitation learning by using optimal
transport to annotate offline datasets with suitable rewards.

##  3 Offline Imitation Learning with Optimal Transport

We consider learning in an episodic, finite-horizon Markov Decision Process
(MDP) $(\mathcal{S},\mathcal{A},p,r,\gamma,p_{0},T)$ where $\mathcal{S}$ is
the state space, $\mathcal{A}$ is the action space, $p$ is the transition
function, $r$ is the reward function, $\gamma$ is the discount factor, $p_{0}$
is the initial state distribution and $T$ is the episode horizon. A policy
$\pi$ is a function from state to a distribution over actions. The goal of RL
is to find policies that maximize episodic return. Running a policy $\pi$ in
the MDP generates a state-action episode/trajectory
$(s_{1},a_{1},s_{2},a_{2},\dots,s_{T})=:\tau$.

We consider the problem of imitation learning purely from offline datasets.
Unlike in the standard RL setting, no explicit reward function is available.
Let $\tau=(s_{1},a_{1},s_{2},a_{2},\dots,s_{T})$ denote an episode of
interaction with the MDP using a policy that selects actions $a_{t}$ at time
steps $t=1,\ldots,T-1$. Instead of a reward function, we have access to a
dataset of _expert_ demonstrations
$\mathcal{D}^{e}=\\{\tau_{e}^{(n)}\\}_{n=1}^{N}$ generated by an expert policy
$\pi_{e}$ and a large dataset of _unlabeled_ trajectories
$\mathcal{D}^{u}=\\{\tau_{\beta}^{(m)}\\}_{m=1}^{M}$ generated by an arbitrary
behavior policy $\pi_{\beta}$. We are interested in learning an offline policy
$\pi$ combining information from the expert demonstrations and unlabeled
experience, without any interaction with the environment. We will address this
problem by using optimal transport, which will provide a way to efficiently
annotate large offline RL datasets with rewards.

###  3.1 Reward Labeling via Wasserstein Distance

Optimal Transport (OT) (Cuturi, 2013; Peyré & Cuturi, 2020) is a principled
approach for comparing probability measures. The (squared) Wasserstein
distance between two discrete measures
$\mu_{x}=\frac{1}{T}\sum^{T}_{t=1}\delta_{x_{t}}$ and
$\mu_{y}=\frac{1}{T^{\prime}}\sum^{T^{\prime}}_{t=1}\delta_{y_{t}}$ is

$\mathcal{W}^{2}(\mu_{x},\mu_{y})=\min_{\mu\in
M}\sum_{t=1}^{T}\sum_{t^{\prime}=1}^{T^{\prime}}c(x_{t},y_{t^{\prime}})\mu_{t,t^{\prime}},$
(1)

where $M=\\{\mu\in\mathbb{R}^{T\times
T^{\prime}}:\mu\mathbf{1}=\frac{1}{T}\mathbf{1},\mu^{T}\mathbf{1}=\frac{1}{T^{\prime}}\mathbf{1}\\}$
is the set of coupling matrices, $c$ is a cost function, and $\delta_{x}$
refers to the Dirac measure for $x$. The optimal coupling $\mu^{*}$ provides
an alignment between the samples in $\mu_{x}$ and $\mu_{y}$. Unlike other
divergence measures (e.g., KL-divergence), the Wasserstein distance is a
metric and it incorporates the geometry of the space.

Let
$\hat{p}_{e}=\frac{1}{T^{\prime}}\sum^{T^{\prime}}_{t=1}\delta_{s_{t}^{e}}$
and $\hat{p}_{\pi}=\frac{1}{T}\sum^{T}_{t=1}\delta_{s_{t}^{\pi}}$ denote the
empirical state distribution of an expert policy $\pi_{e}$ and behavior policy
$\pi$ respectively. Then the (squared) Wasserstein distance

$\mathcal{W}^{2}(\hat{p}_{\pi},\hat{p}_{e})=\min_{\mu\in
M}\sum_{t=1}^{T}\sum_{t^{\prime}=1}^{T^{\prime}}c(s_{t}^{\pi},s^{e}_{t^{\prime}})\mu_{t,t^{\prime}}$
(2)

can be used to measure the distance between expert policy and behavior policy.
Let $\mu^{*}$ denote the optimal coupling for the optimization problem above,
then eq. 2 provides a reward signal

$r_{\text{ot}}(s_{t}^{\pi})=-\sum_{t^{\prime}=1}^{T^{\prime}}c(s_{t}^{\pi},s_{t^{\prime}}^{e})\mu^{*}_{t,t^{\prime}},$
(3)

which can be used for learning policy $\pi$ in an imitation learning setting.

Figure 2: Illustration of the computations performed by OTR. In this example,
we consider an MDP with a two-dimensional state-space ($|\mathcal{S}|=2$). We
have two empirical state distributions from an expert $\hat{p}_{e}$ with
samples $\\{s_{t^{\prime}}^{e}\\}_{t^{\prime}=1}^{T^{\prime}}$ (${\circ}$) and
policy $\hat{p}_{\pi}$ with samples $\\{s_{t}^{\pi}\\}_{t=1}^{T}$
(${\color[rgb]{0.35,0.87,1}\definecolor[named]{pgfstrokecolor}{rgb}{0.35,0.87,1}\pgfsys@color@cmyk@stroke{0.65}{0.13}{0}{0}\pgfsys@color<EMAIL_ADDRESS>as denoted by points in the leftmost figure. OTR assigns rewards
$r_{\text{ot}}$ (blue) to each sample in the policy’s empirical state
distribution as follows: (i) Compute the pairwise cost matrix $C$ (purple)
between expert trajectories and trajectories generated by behavior policy;
(ii) Solve for the optimal coupling matrix $\mu$ (green) between
$\hat{p}_{\pi}$ and $\hat{p}_{e}$; (iii) Compute the reward for $s_{t}^{\pi}$
as $r_{\text{ot}}(s_{t}^{\pi})=-C_{t}^{T}\mu^{*}_{t}$. Consider for example a
state $s_{t}^{\pi}\in\hat{p}_{\pi}$; the row $C_{t}$ in the cost matrix
corresponds to the costs between $s_{t}^{\pi}$ and
$\\{s_{t^{\prime}}^{e}\\}_{t^{\prime}=1}^{T^{\prime}}$. $\mu^{*}_{t}$
represents the optimal coupling between $s_{t}^{\pi}$ and the expert samples.
The optimal coupling moves most of the probability mass to $s^{e}_{3}$ and a
small fraction of the mass to $s^{e}_{4}$ (green lines in the leftmost
figure).

###  3.2 Imitation Learning Using Reward Labels From Optimal Transport

We leverage the reward function from eq. 3 to annotate the unlabeled dataset
with reward signals. Computing the optimal alignment between the expert
demonstration with trajectories in the unlabeled dataset allows us to assign a
reward for each step in the unlabeled trajectory. Figure 2 illustrates the
computation performed by OTR to annotate an unlabeled dataset with rewards
using demonstrations from an expert.

1

Input: unlabeled dataset $\mathcal{D}^{u}$, expert dataset $\mathcal{D}^{e}$

Output: labeled dataset $\mathcal{D}^{\text{label}}$

2 $\mathcal{D}^{\text{label}}\leftarrow\emptyset$

3 foreach  _ $\tau^{(m)}$ in $\mathcal{D}^{u}$_ do // Label each episode in
the unlabeled dataset

$C^{(m)},\mu^{*(m)}\leftarrow\textnormal{{SolveOT}}(\mathcal{D}^{e},\tau^{(m)})$

// Compute the optimal alignment with eq. 2

4 for  _$t$ = $1$ to $T$_ do

$r_{\text{OT}}(s^{(m)}_{t})\leftarrow-\sum_{t^{\prime}=1}^{T^{\prime}}C^{(m)}_{t,t^{\prime}}\mu^{*(m)}_{t,t^{\prime}}$

// Compute the per-step rewards with eq. 3

5

6 end for

$\mathcal{D}^{\text{label}}\leftarrow\mathcal{D}^{\text{label}}\cup(s_{1}^{(m)},a_{1}^{(m)},r_{1}^{\text{OT}},\dots,s_{T}^{(m)})$

// Append labeled episode

7

8 end foreach

9return _$\mathcal{D}^{\text{label}}$_

Algorithm 1 Pseudo-code for Optimal Transport Reward labeling (OTR)

The pseudo-code for our approach is given in algorithm 1. OTR takes the
unlabeled dataset $\mathcal{D}^{u}$ and expert demonstration $\mathcal{D}^{e}$
as input. For each unlabeled trajectory $\tau^{(m)}\in\mathcal{D}^{u}$, OTR
solves the optimal transport problem for each, obtaining the cost matrix
$C^{(m)}$ and optimal alignment $\mu^{*(m)}$ (algorithm 1). OTR then computes
the per-step reward label following eq. 3 (algorithm 1). The reward-annotated
trajectories are then combined, forming a reward-labeled dataset
$\mathcal{D}^{\text{label}}$.

Solving eq. 2 requires solving the OT problem to obtain the optimal coupling
matrix $\mu^{*}$. This amounts to solving a linear program (LP), which may be
prohibitively expensive with a standard LP solver. In practice, we solve the
entropy-regularized OT problem with Sinkhorn’s algorithm (Cuturi, 2013). We
leverage the Sinkhorn solver in OTT-JAX (Cuturi et al., 2022) for this
computation. Once OTR has annotated the unlabeled offline dataset with
intrinsic rewards, we can use an offline RL algorithm for learning a policy.
Since we are working in the pure offline setting, it is important to use an
offline RL algorithm that can minimize the distribution shifts typically
encountered in the offline setting.

Unlike prior works that compute rewards using online samples (Dadashi et al.,
2022; Cohen et al., 2021), we compute the rewards entirely offline, prior to
running offline RL training, avoiding the need to modify any part of the
downstream offline RL pipeline. Therefore, our approach can be combined with
any offline RL algorithms, providing dense reward annotations that are
required by the downstream algorithms. Figure 1 illustrates the entire
pipeline of using OTR for relabeling and running an offline RL algorithm using
the reward annotated datasets.

Compared to previous work that aims at solving offline imitation learning with
a single algorithm, OTR focuses on generating high-quality reward annotations
for downstream offline RL algorithms. In addition, our approach enjoys several
advantages:

  * •

Our approach does not require training separate reward models or
discriminators, which may incur higher runtime overhead. By not having to
train a separate parametric model, we avoid hyper-parameter tuning on the
discriminator network architectures.

  * •

Unlike other approaches, such as GAIL or DemoDICE, our approach does not
require solving a minimax optimization problem, which can suffer from training
instability (Orsini et al., 2021).

  * •

Our approach is agnostic to the offline RL methods for learning the policy
since OTR computes reward signals independently of the offline RL algorithm.

##  4 Experiments

Figure 3: Benchmark tasks: D4RL Locomotion, Antmaze, and Adroit.

In this section, we evaluate OTR on D4RL Locomotion, Antmaze, and Adroit
benchmark tasks (see also Figure 3) with the goal of answering the following
research questions:

  1. 1.

Can OTR recover the performance of offline RL algorithms that has access to a
well-engineered reward function (i.e., ground-truth rewards provided by the
environment)?

  2. 2.

Can OTR handle unlabeled datasets with behaviors of unknown and mixed quality?

  3. 3.

How does OTR perform with a varying number of expert demonstrations?

  4. 4.

How does OTR compare with previous work on offline imitation learning in terms
of performance and runtime complexity?

We demonstrate that OTR can be effectively combined with an offline RL
algorithm to learn policies from a large dataset of unlabeled episodes and a
small number of high-quality demonstrations. Since OTR is only a method for
reward learning, it can be combined with any offline RL algorithm that
requires reward-annotated data for offline learning. In this paper, we combine
OTR with the Implicit $Q$-Learning (IQL) algorithm (Kostrikov et al., 2022c).

###  4.1 Setup

We evaluate the performance of OTR+IQL on the D4RL locomotion benchmark (Fu et
al., 2021). We start by evaluating OTR on three environments (HalfCheetah-v2,
Walker-v2, Hopper-v2) from the OpenAI Gym MuJoCo locomotion tasks. For each
environment, we use the medium-v2, medium-replay-v2 and medium-expert-v2
datasets to construct the expert demonstrations and the unlabeled dataset. For
the expert demonstrations, we choose the best episodes from the D4RL dataset
based on the episodic return.222In practice, the expert demonstrations can be
provided separately; we only select the expert demonstration in this way for
ease of evaluation. To obtain the unlabeled dataset, we discard the original
reward information in the dataset. We then run OTR to label the dataset based
on the optimal coupling between the unlabeled episodes and the chosen expert
demonstrations. Afterward, we proceed with running the offline RL algorithm.

For OTR, we follow the recommendation by Cohen et al. (2021) and use the
cosine distance as the cost function. When there are more than one episode of
expert demonstrations, we compute the optimal transport with respect to each
episode independently and use the rewards from the expert trajectory that
gives the best episodic return. Similar to (Dadashi et al., 2022; Cohen et
al., 2021), we squash the rewards computed by algorithm 1 with an exponential
function $s(r)=\alpha\exp(\beta r)$. This has the advantage of ensuring that
the rewards consumed by the offline RL algorithm have an appropriate range
since many offline RL algorithms can be sensitive to the scale of the reward
values. We refer to section A.1 for additional experimental details and
hyperparameters.

##### Implementation

We implement OTR in JAX (Bradbury et al., 2018). For computing the optimal
coupling, we use OTT-JAX (Cuturi et al., 2022), a library for optimal
transport that includes a scalable and efficient implementation of the
Sinkhorn algorithm that can leverage accelerators, such as GPU or TPU, for
speeding up computations. Our IQL implementation is adapted from (Kostrikov et
al., 2022c)333https://github.com/ikostrikov/implicit_q_learning, and we set
all hyperparameters to the ones recommended in the original paper. All of our
algorithms and baselines are implemented in Acme (Hoffman et al., 2022).

Our implementation is computationally efficient444An efficient implementation
of OTR is facilitated by JAX, which includes useful functionality that allows
us to easily parallelize computations. Concretely, we JIT-compile the
computation of rewards for one episode and further leverage the vmap function
to compute the optimal coupling between an unlabeled episode with all of the
expert episodes in parallel. Efficiently parallelizing the computation of the
optimal coupling requires that all the episodes share the same length. This is
necessary both for parallelizing the computation across multiple expert
demonstrations as well as for avoiding recompilation by XLA due to changes in
the shape of the input arrays. To achieve high throughput for datasets with
varying episodic length, we pad all observations to the maximum episode length
allowed by the environment (which is $1000$ for the OpenAI Gym environments)
but set the weights of the observations to zero. Padding the episodes this way
does not change the solution to the optimal coupling problem. Note that
padding means that a 1M transition dataset may create more than $1000$
episodes of experience, in this case, the runtime for our OTR implementation
may be higher effectively due to having to process a larger number of padded
episodes., requiring only about $1$ minute to label a dataset with $1$ million
transitions (or $1000$ episodes of length $1000$)555Runtime measured on
halfcheetah-medium-v2 with an NVIDIA GeForce RTX 3080 GPU.. For larger-scale
problems, OTR can be scaled up further by processing the episodes in the
dataset in parallel. Our implementation of OTR and re-implementation of
baselines are computationally efficient. Even so, the training time for IQL is
about $20$ minutes, so that OTR adds a relatively small amount of overhead for
reward annotation to an existing offline RL algorithm.

##### Baselines

We compare OTR+IQL with the following baselines:

  * •

IQL (oracle): this is the original Implicit $Q$-learning (Kostrikov et al.,
2022c) algorithm using the ground-truth rewards provided by the D4RL datasets.

  * •

DemoDICE: an offline imitation learning algorithm proposed by Kim et al.
(2022). DemoDICE was found to perform better than previous imitation learning
algorithms (e.g., ValueDICE (Kostrikov et al., 2022b)) that can also (in
principle) work in the pure offline setting. We ran the original
implementation666https://github.com/geon-hyeong/imitation-dice under the same
experimental setting as we used for the other algorithms in the paper.

  * •

ORIL: a reward function learning algorithm from (Zolna et al., 2020). ORIL
learns a reward function by contrasting the expert demonstrations with the
unlabeled episodes. The learned reward function is then used for labeling the
unlabeled dataset. We implement ORIL in a comparable setting to the other
baselines.

  * •

UDS: we keep the rewards from the expert demonstrations and relabel all
rewards from the unlabeled datasets with the minimum rewards from the
environment. This was found to perform well in (Yu et al., 2022). This also
resembles online imitation learning methods such as SQIL (Reddy et al., 2022)
or offline RL algorithms, such as COG (Singh et al., 2021).

Since UDS and ORIL are also agnostic about the underlying offline RL algorithm
used, we combined these algorithms with IQL so that we can focus on comparing
the performance difference due to different approaches used in generating
reward labels.

For all algorithms, we repeat experiments with $10$ random seeds and report
the mean and standard deviation of the normalized performance of the last $10$
episodes of evaluation. We compare all algorithms by using either $K=1$ or
$K=10$ expert demonstrations. Obtaining the results on the locomotion datasets
took approximately $500$ GPU hours.

###  4.2 Results

Dataset | BC | 10%BC | IQL (oracle) | DemoDICE | IQL+ORIL | IQL+UDS | OTR+IQL (ours)
---|---|---|---|---|---|---|---
halfcheetah-medium-v2 | 42.6 | 42.5 | 47.4 $\pm$ 0.2 | 42.5 $\pm$ 1.7 | 49.0 $\pm$ 0.2 | 42.4 $\pm$ 0.3 | 43.3 $\pm$ 0.2
hopper-medium-v2 | 52.9 | 56.9 | 66.2 $\pm$ 5.7 | 55.1 $\pm$ 3.3 | 47.0 $\pm$ 4.0 | 54.5 $\pm$ 3.0 | 78.7 $\pm$ 5.5
walker2d-medium-v2 | 75.3 | 75.0 | 78.3 $\pm$ 8.7 | 73.4 $\pm$ 2.6 | 61.9 $\pm$ 6.6 | 68.9 $\pm$ 6.2 | 79.4 $\pm$ 1.4
halfcheetah-medium-replay-v2 | 36.6 | 40.6 | 44.2 $\pm$ 1.2 | 38.1 $\pm$ 2.7 | 44.1 $\pm$ 0.6 | 37.9 $\pm$ 2.4 | 41.3 $\pm$ 0.6
hopper-medium-replay-v2 | 18.1 | 75.9 | 94.7 $\pm$ 8.6 | 39.0 $\pm$ 15.4 | 82.4 $\pm$ 1.7 | 49.3 $\pm$ 22.7 | 84.8 $\pm$ 2.6
walker2d-medium-replay-v2 | 26.0 | 62.5 | 73.8 $\pm$ 7.1 | 52.2 $\pm$ 13.1 | 76.3 $\pm$ 4.9 | 17.7 $\pm$ 9.6 | 66.0 $\pm$ 6.7
halfcheetah-medium-expert-v2 | 55.2 | 92.9 | 86.7 $\pm$ 5.3 | 85.8 $\pm$ 5.7 | 87.5 $\pm$ 3.9 | 63.0 $\pm$ 5.7 | 89.6 $\pm$ 3.0
hopper-medium-expert-v2 | 52.5 | 110.9 | 91.5 $\pm$ 14.3 | 92.3 $\pm$ 14.2 | 29.7 $\pm$ 22.2 | 53.9 $\pm$ 2.5 | 93.2 $\pm$ 20.6
walker2d-medium-expert-v2 | 107.5 | 109.0 | 109.6 $\pm$ 1.0 | 106.9 $\pm$ 1.9 | 110.6 $\pm$ 0.6 | 107.5 $\pm$ 1.7 | 109.3 $\pm$ 0.8
locomotion-v2-total | 466.7 | 666.2 | 692.4 | 585.3 | 588.5 | 494.9 | 685.5
runtime | 10m | 10m | 20m | 100m* | 30m | 20m | 22m

  * *

The runtime is measured with the original PyTorch implementation.

Table 1: D4RL performance comparison between IQL with ground-truth rewards and
OTR+IQL with a single expert demonstration ($K=1$). We report mean $\pm$
standard deviation per task and aggregate performance and highlight near-
optimal performance in bold and extreme negative outliers in red. OTR+IQL is
the only algorithm that performs consistently well across all domains.

##### MuJoCo Locomotion

Table 1 compares the performance between OTR+IQL with the other baselines.
Overall, OTR+IQL performs best compared with the other baselines in terms of
aggregate score over all of the datasets we used, recovering the performance
of IQL with ground-truth rewards provided by the dataset. While we found that
other baselines can perform well on some datasets, the performance is not
consistent across the entire dataset and can deteriorate significantly on some
datasets. In contrast, OTR+IQL is the only method that consistently performs
well for all datasets of different compositions.

##### Runtime

Despite applying optimal transport, we found that with a GPU-accelerated
Sinkhorn solver (Cuturi et al., 2022), combined with our efficient
implementation in JAX, OTR achieves a faster runtime compared to algorithms
that learn additional neural networks as discriminators (DemoDICE (Kim et al.,
2022)) or reward models (ORIL (Zolna et al., 2020)). For methods that learn a
neural network reward function, an overhead of at least $10$ minutes is
incurred, whereas OTR only incurs approximately $2$ minutes of overheads when
compared with the same amount of expert demonstrations.

##### Effect of the number of demonstrations

locomotion-v2-total | $K=1$ | $K=10$
---|---|---
DemoDICE | 585.3 | 589.3
IQL+ORIL | 588.5 | 618.3
IQL+UDS | 494.9 | 575.8
OTR+IQL | 685.5 | 694.3
IQL (oracle) |  | 692.4
Table 2: Aggregate performances of different reward labeling algorithms with
different numbers of expert demonstrations. OTR is the only algorithm that
leads to an offline RL performance close to using ground-truth rewards.

We investigate if the performance of the baselines can be improved by
increasing the number of expert demonstrations used. Table 2 compares the
aggregate performance on the locomotion datasets between OTR and the baselines
when we increase the number of demonstrations from $K=1$ to $K=10$. DemoDICE’s
performance improves little with the additional amount of expert demonstrates.
While ORIL and UDS enjoy a relatively larger improvement, they are still
unable to match the performance of IQL (oracle) or OTR in terms of aggregate
performance despite using the same offline RL backbone. OTR’s performance is
close to IQL (oracle) even when $K=1$ and matches the performance of IQL
(oracle) with $K=10$.

##### Antmaze & Adroit

We additionally evaluate OTR+IQL on the antmaze-v0 and adroit-v0 datasets.
Table 3 shows that OTR+IQL again recovers the performance of IQL with ground-
truth rewards. This suggests that OTR+IQL can learn from datasets with diverse
behavior and human demonstrations even without ground-truth reward annotation;
additional results in section A.2.

Dataset | IQL (oracle) | OTR+IQL
---|---|---
antmaze-large-diverse-v0 | 47.5 $\pm$ 9.5 | 45.5 $\pm$ 6.2
antmaze-large-play-v0 | 39.6 $\pm$ 5.8 | 45.3 $\pm$ 6.9
antmaze-medium-diverse-v0 | 70.0 $\pm$ 10.9 | 70.4 $\pm$ 4.8
antmaze-medium-play-v0 | 71.2 $\pm$ 7.3 | 70.5 $\pm$ 6.6
antmaze-umaze-diverse-v0 | 62.2 $\pm$ 13.8 | 68.9 $\pm$ 13.6
antmaze-umaze-v0 | 87.5 $\pm$ 2.6 | 83.4 $\pm$ 3.3
antmaze-v0-total | 378.0 | 384.0

Dataset | IQL (oracle) | OTR+IQL
---|---|---
door-cloned-v0 | 1.60 | 0.01 $\pm$ 0.01
door-human-v0 | 4.30 | 5.92 $\pm$ 2.72
hammer-cloned-v0 | 2.10 | 0.88 $\pm$ 0.30
hammer-human-v0 | 1.40 | 1.79 $\pm$ 1.43
pen-cloned-v0 | 37.30 | 46.87 $\pm$ 20.85
pen-human-v0 | 71.50 | 66.82 $\pm$ 21.18
relocate-cloned-v0 | -0.20 | -0.24 $\pm$ 0.03
relocate-human-v0 | 0.10 | 0.11 $\pm$ 0.10
adroit-v0-total | 118.1 | 122.16

Table 3: Performance of OTR+IQL on antmaze and adroit with a single expert
demonstration. Similar to the results on locomotion, OTR+IQL recovers the
performance of offline RL with ground-truth rewards. We report mean $\pm$
standard deviation per task and aggregate performance.

(a) Ground-truth return vs labeled return. (b) Top trajectories selected by
OTR.

Figure 4: Visualization of rewards predicted by OTR and baselines. 4(a)
Qualitative differences between the rewards predicted by OTR, ORIL and UDS on
hopper-medium-v2. 4(b) Visualization of top trajectories selected by OTR on
antmaze-medium-play-v0. Left: Expert demonstrations. Right: ranking of
trajectories according to rewards per step computed by OTR. Trajectories with
lighter colors have higher rewards per step.

##### Qualitative comparison of the reward predictions

Figure 4(a) provides a qualitative comparison of the reward predicted by OTR,
ORIL, and UDS. UDS annotates all transitions in the unlabeled dataset with the
minimum reward from the environment. Thus, the episodes with non-zero rewards
are expert demonstrations. This means that UDS is unable to distinguish
between episodes in the unlabeled datasets. Compared to reward learning
algorithms, such as ORIL, OTR’s reward prediction more strongly correlates
with the ground-truth rewards from the environment, which is a good
precondition for successful policy learning by downstream offline RL
algorithm. We also evaluate OTR’s reward prediction on more diverse datasets,
such as those in antmaze. Figure 4(b) shows the expert demonstrations we used
in antmaze-medium-play-v0 (left) and the trajectories that received the best
OTR reward labels in the unlabeled dataset (right). OTR correctly assigns more
rewards to trajectories that are closer to the expert demonstrations.

##### Summary

Our empirical evaluation demonstrates that OTR+IQL can recover the performance
of offline RL with ground-truth rewards given only a single episode of expert
demonstration. Compared to previous work, it achieves better performance even
with a larger number of demonstrations. Our qualitative comparison shows that
OTR assigns better reward estimates that correlate strongly with a hand-
crafted reward function.

##  5 Discussion

OTR can be effective in augmenting unlabeled datasets with rewards for use by
downstream offline RL algorithms (see section 4). Compared to prior imitation
learning and reward learning approaches, OTR enjoys better and more robust
performance on the benchmarks we have considered in this paper. Furthermore,
our approach is easy to implement. Most of the complexity of our approach
arises from solving the optimal transport problem. However, there are
libraries for solving OT efficiently in different frameworks or programming
languages777For example, see Python Optimal Transport (POT) (Flamary et al.,
2021), which supports PyTorch, JAX, or TensorFlow..

Unlike prior works, such as ValueDICE or DemoDICE, we split offline imitation
learning into the two distinct phases of reward modeling via OTR and a
subsequent offline RL. This improves modularity, allows for improvements in
the two phases to be made independently, and adds flexibility. However, this
modularity means that a practitioner needs to make a separate choice for the
downstream offline RL algorithm.

We believe that OTR is most useful in situations where providing ground-truth
rewards is difficult but providing good demonstrations and unlabeled data is
feasible. However, it will not be feasible in cases where collecting expert
demonstrations is difficult.

The Wasserstein distance formulation we used can be extended to perform cross-
domain imitation learning by using the Gromov–Wasserstein distance to align
expert demonstrations and offline trajectories from different spaces
(Fickinger et al., 2022).

##  6 Conclusion

We introduced Optimal Transport Reward labeling (OTR), a method for adding
reward labels to an offline dataset, given one or more expert demonstrations.
OTR computes Wasserstein distances between expert demonstrations and
trajectories in a dataset without reward labels, which then are turned into a
reward signal. The reward-annotated offline dataset can then be used by an(y)
offline RL algorithm to determine good policies. OTR adds minimal overhead to
existing offline RL algorithms while providing the same performance as
learning with a pre-specified reward function.

#### Acknowledgments

Yicheng Luo is supported by the UCL Overseas Research Scholarship (UCLORS).
Samuel Cohen is supported by the UKRI AI Centre for Doctoral Training in
Foundational Artificial Intelligence (EP/S021566/1).

## References

  * Abbeel & Ng (2004) Pieter Abbeel and Andrew Y. Ng.  Apprenticeship Learning via Inverse Reinforcement Learning.  In _International Conference on Machine Learning_ , 2004.
  * Atkeson & Schaal (1997) Christopher G. Atkeson and Stefan Schaal.  Robot Learning From Demonstration.  In _International Conference on Machine Learning_ , 1997.
  * Bradbury et al. (2018) James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: Composable Transformations of Python+NumPy Programs, 2018.  URL http://github.com/google/jax.
  * Calinon et al. (2007) Sylvain Calinon, Florent Guenter, and Aude Billard.  On Learning, Representing, and Generalizing a Task in a Humanoid Robot.  _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , 2007.
  * Cohen et al. (2021) Samuel Cohen, Brandon Amos, Marc Peter Deisenroth, Mikael Henaff, Eugene Vinitsky, and Denis Yarats.  Imitation Learning from Pixel Observations for Continuous Control.  2021\.  URL https://openreview.net/forum?id=JLbXkHkLCG6.
  * Cuturi (2013) Marco Cuturi.  Sinkhorn Distances: Lightspeed Computation of Optimal Transport.  In _Advances in Neural Information Processing Systems_ , 2013\.
  * Cuturi et al. (2022) Marco Cuturi, Laetitia Meng-Papaxanthos, Yingtao Tian, Charlotte Bunne, Geoff Davis, and Olivier Teboul.  Optimal Transport Tools (OTT): A JAX Toolbox for all things Wasserstein.  arXiv:2201.12324, 2022.
  * Dadashi et al. (2022) Robert Dadashi, Leonard Hussenot, Matthieu Geist, and Olivier Pietquin.  Primal Wasserstein Imitation Learning.  In _International Conference on Learning Representations_ , 2022.
  * Elkan & Noto (2008) Charles Elkan and Keith Noto.  Learning Classifiers from Only Positive and Unlabeled Data.  In _International Conference on Knowledge Discovery and Data Mining_ , 2008.
  * Englert et al. (2013) Peter Englert, Alexandros Paraschos, Jan Peters, and Marc Peter Deisenroth.  Model-based Imitation Learning by Probabilistic Trajectory Matching.  In _IEEE International Conference on Robotics and Automation_ , 2013.
  * Fickinger et al. (2022) Arnaud Fickinger, Samuel Cohen, Stuart Russell, and Brandon Amos.  Cross-Domain Imitation Learning via Optimal Transport.  In _International Conference on Learning Representations_ , 2022.
  * Flamary et al. (2021) Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T. H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer.  POT: Python Optimal Transport.  _Journal of Machine Learning Research_ , 2021.
  * Fu et al. (2021) Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.  D4RL: Datasets for Deep Data-Driven Reinforcement Learning.  arXiv:2004.07219, 2021.
  * Fujimoto & Gu (2021) Scott Fujimoto and Shixiang Gu.  A Minimalist Approach to Offline Reinforcement Learning.  In _Advances in Neural Information Processing Systems_ , 2021\.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative Adversarial Nets.  In _Advances in Neural Information Processing Systems_ , 2014\.
  * Ho & Ermon (2016) Jonathan Ho and Stefano Ermon.  Generative Adversarial Imitation Learning.  In _Advances in Neural Information Processing Systems_ , 2016\.
  * Hoffman et al. (2022) Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Stańczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac-Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas.  Acme: A Research Framework for Distributed Reinforcement Learning.  arXiv:2006.00979, 2022.
  * Kim et al. (2022) Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim.  DemoDICE: Offline Imitation Learning with Supplementary Imperfect Demonstrations.  In _International Conference on Learning Representations_ , 2022.
  * Kostrikov et al. (2022a) Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson.  Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning.  In _International Conference on Learning Representations_ , 2022a.
  * Kostrikov et al. (2022b) Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson.  Imitation Learning via Off-Policy Distribution Matching.  In _International Conference on Learning Representations_ , 2022b.
  * Kostrikov et al. (2022c) Ilya Kostrikov, Ashvin Nair, and Sergey Levine.  Offline Reinforcement Learning with Implicit Q-Learning.  In _International Conference on Learning Representations_ , 2022c.
  * Kumar et al. (2020) Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative Q-Learning for Offline Reinforcement Learning.  In _Advances in Neural Information Processing Systems_ , 2020\.
  * Lange et al. (2012) Sascha Lange, Thomas Gabel, and Martin Riedmiller.  Batch Reinforcement Learning.  In _Reinforcement Learning_. Springer Berlin Heidelberg, 2012\.
  * Levine et al. (2020) Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.  arXiv:2005.01643, 2020.
  * Ng & Russell (2000) Andrew Y. Ng and Stuart J. Russell.  Algorithms for Inverse Reinforcement Learning.  In _International Conference on Machine Learning_ , 2000.
  * Orsini et al. (2021) Manu Orsini, Anton Raichuk, Leonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowicz.  What Matters for Adversarial Imitation Learning?  In _Advances in Neural Information Processing Systems_ , 2021\.
  * Peyré & Cuturi (2020) Gabriel Peyré and Marco Cuturi.  Computational Optimal Transport.  arXiv:1803.00567, 2020.
  * Pomerleau (1988) Dean A. Pomerleau.  ALVINN: An Autonomous Land Vehicle in a Neural Network.  In _Advances in Neural Information Processing Systems_ , 1988\.
  * Reddy et al. (2022) Siddharth Reddy, Anca D. Dragan, and Sergey Levine.  SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards.  In _International Conference on Learning Representations_ , 2022.
  * Singh et al. (2021) Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine.  Chaining Behaviors from Data with Model-Free Reinforcement Learning.  In _Conference on Robot Learning_ , 2021.
  * Wang et al. (2020) Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas.  Critic Regularized Regression.  In _Advances in Neural Information Processing Systems_ , 2020\.
  * Xiao et al. (2019) Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong Linh.  Wasserstein Adversarial Imitation Learning.  arXiv:1906.08113, 2019.
  * Yu et al. (2022) Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine.  How to Leverage Unlabeled Data in Offline Reinforcement Learning.  In _International Conference on Machine Learning_ , 2022.
  * Zolna et al. (2020) Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed.  Offline Learning from Demonstrations and Unlabeled Experience.  arXiv:2011.13885, 2020.

##  Appendix A Appendix

###  A.1 Hyperparameters

Table 4 lists the hyperparameters used by OTR and IQL on the locomotion
datasets. For Antmaze and Adroit, unless otherwise specified by table 5 or
table 6, the hyperparameters follows from those used in the locomotion
datasets.

The IQL hyperparameters are kept the same as those used in (Kostrikov et al.,
2022c). Note that IQL rescales the rewards in the dataset so that the same set
of hyperparameters can be used for datasets of different qualities. Since OTR
computes rewards offline, we also apply reward scaling as in IQL. For the
locomotion datasets, the rewards are rescaled by
$\frac{1000}{\text{max\\_return}-\text{min\\_return}}$ while for antmaze we
subtract $2$ to the rewards computed by OTR. The reward processing in antmaze
is different from the one used by the original IQL paper (which subtracts $1$)
since the rewards computed by OTR have a different range.

The squashing function used by OTR is based on the one used in (Dadashi et
al., 2022). The antmaze squashing differs slightly from the one used in
locomotion and adroit due to use of an earlier configuration. In practice,
this should have minimal effect on the performance.

| Hyperparameter | Value
---|---|---
| Discount | $0.99$
Network Architectures | Hidden layers | $(256,256)$
Dropout | none
| Network initialization | orthogonal
IQL | Optimizer | Adam
Policy learning rate |  $3e^{-4}$, cosine decay to $0$
| Critic learning rate | $3e^{-4}$
| Value learning rate | $3e^{-4}$
| Target network update rate | $5e^{-3}$
| Temperature | 3.0
| Expectile | 0.7
OTR | Episode length $T$ | $1000$
Cost function | cosine
Squashing function | $s(r)=5.0\cdot\exp(5.0\cdot T\cdot r/|\mathcal{A}|)$
Table 4: OTR hyperparameters for D4RL Locomotion. | Hyperparameter | Value
---|---|---
IQL | Temperature | 10.0
Expectile | 0.9
OTR | Squashing function | $s(r)=5.0\cdot\exp(T\cdot r)$
Table 5: OTR hyperparameters for D4RL Antmaze. | Hyperparameter | Value
---|---|---
Network Architectures | Dropout | 0.1
IQL | Temperature | 0.5
Expectile | 0.7
Table 6: OTR hyperparameters for D4RL Adroit.

###  A.2 Additional Experimental Results on Adroit and Antmaze

We evaluate OTR on additional datasets from the antmaze and adroit domains
with varying number of expert demonstrations. The results are presented in
table 7 and table 8. OTR consistently recovers the performance of IQL with
ground-truth rewards on these datasets, largely independent of the number $K$
of expert demonstrations provided.

|  | $K=1$ | $K=10$
---|---|---|---
Dataset | IQL (oracle) | OTR+IQL | OTR+IQL
door-cloned-v0 | 1.60 | 0.01$\pm$ | 0.01 | 0.01$\pm$ | 0.01
door-human-v0 | 4.30 | 5.92$\pm$ | 2.72 | 4.15$\pm$ | 2.08
hammer-cloned-v0 | 2.10 | 0.88$\pm$ | 0.30 | 1.31$\pm$ | 0.70
hammer-human-v0 | 1.40 | 1.79$\pm$ | 1.43 | 1.36$\pm$ | 0.22
pen-cloned-v0 | 37.30 | 46.87$\pm$ | 20.85 | 42.68$\pm$ | 24.98
pen-human-v0 | 71.50 | 66.82$\pm$ | 21.18 | 69.41$\pm$ | 21.50
relocate-cloned-v0 | -0.20 | -0.24$\pm$ | 0.03 | -0.24$\pm$ | 0.03
relocate-human-v0 | 0.10 | 0.11$\pm$ | 0.10 | 0.10$\pm$ | 0.07
adroit-v0-total | 118.1 | 122.16 | 118.78
Table 7: OTR+IQL Results on Adroit. The standard deviations for IQL (oracle) are not available from (Kostrikov et al., 2022c). Dataset | IQL (oracle) | OTR+IQL ($K=1$) | OTR+IQL ($K=10$)
---|---|---|---
antmaze-large-diverse-v0 | 47.5$\pm$ | 9.5 | 45.5$\pm$ | 6.2 | 50.7$\pm$ | 6.9
antmaze-large-play-v0 | 39.6$\pm$ | 5.8 | 45.3$\pm$ | 6.9 | 51.2$\pm$ | 7.1
antmaze-medium-diverse-v0 | 70.0$\pm$ | 10.9 | 70.4$\pm$ | 4.8 | 70.5$\pm$ | 6.9
antmaze-medium-play-v0 | 71.2$\pm$ | 7.3 | 70.5$\pm$ | 6.6 | 72.7$\pm$ | 6.2
antmaze-umaze-diverse-v0 | 62.2$\pm$ | 13.8 | 68.9$\pm$ | 13.6 | 64.4$\pm$ | 18.2
antmaze-umaze-v0 | 87.5$\pm$ | 2.6 | 83.4$\pm$ | 3.3 | 88.7$\pm$ | 3.5
antmaze-v0-total | 378.0 | 384.0 | 398.2
Table 8: OTR+IQL Results on Antmaze.

###  A.3 Combining OTR with Different Offline RL Algorithms

In the main experiments, we evaluated OTR by pairing it with the IQL
algorithm. In this section, we investigate if OTR can recover the performance
of a different offline RL algorithm (TD3-BC) (Fujimoto & Gu, 2021) using
ground-truth rewards. We observe that (i) the performance from OTR+TD3-BC
mostly matches those using the ground-truth rewards; (ii) the performance is
fairly robust to the choice of the number of expert trajectories ($K=1$ and
$K=10$ many expert demonstrations provide comparable performance). However,
There are more variances on some datasets (e.g., halfcheetah-medium-
expert-v2). Nevertheless, the differences are smaller compared to the
baselines and OTR+TD3-BC still performs better than the baselines presented in
section 4 in terms of the aggregate performance.

Dataset | TD3-BC (oracle) | OTR+TD3-BC (K=1) | OTR+TD3-BC (K=10)
---|---|---|---
halfcheetah-medium-expert-v2 | 93.5$\pm$ | 2.0 | 74.8$\pm$ | 20.1 | 71.6$\pm$ | 23.1
halfcheetah-medium-replay-v2 | 44.4$\pm$ | 0.8 | 39.4$\pm$ | 1.3 | 38.9$\pm$ | 1.5
halfcheetah-medium-v2 | 48.0$\pm$ | 0.7 | 42.6$\pm$ | 1.0 | 42.7$\pm$ | 1.1
hopper-medium-expert-v2 | 100.2$\pm$ | 20.0 | 103.2$\pm$ | 13.9 | 98.9$\pm$ | 19.7
hopper-medium-replay-v2 | 64.8$\pm$ | 25.5 | 74.9$\pm$ | 28.8 | 80.2$\pm$ | 23.1
hopper-medium-v2 | 60.7$\pm$ | 12.5 | 66.4$\pm$ | 10.3 | 69.8$\pm$ | 13.9
walker2d-medium-expert-v2 | 109.5$\pm$ | 0.5 | 109.0$\pm$ | 0.6 | 108.8$\pm$ | 0.8
walker2d-medium-replay-v2 | 87.4$\pm$ | 8.4 | 69.7$\pm$ | 16.4 | 67.4$\pm$ | 20.6
walker2d-medium-v2 | 83.7$\pm$ | 5.3 | 76.9$\pm$ | 5.4 | 78.0$\pm$ | 2.6
locomotion-v2-total | 692.3 | 656.9 | 656.4
Table 9: OTR+TD3-BC Results on MuJoCo.

###  A.4 Importance of Using the Optimal Transport Plan

In the main experiments, we compute the rewards based on the optimal coupling
computed by the Sinkhorn solver. The optimal transport plan is sparse and
transports most of the probability masses to only a few expert samples. In
this section, we investigate what happens if we use a suboptimal transport
plan where each sample from the policy’s trajectory is transported equally to
each sample in the expert’s trajectory. In this case, the reward function
essentially boils down to computing the average costs with respect to all of
the states in the expert’s trajectory.

Dataset | OTR+IQL | OTR+IQL (Uniform Plan)
---|---|---
halfcheetah-medium-v2 | 43.3$\pm$ | 0.2 | 43.5$\pm$ | 0.3
hopper-medium-v2 | 78.7$\pm$ | 5.5 | 80.5$\pm$ | 2.3
walker2d-medium-v2 | 79.4$\pm$ | 1.4 | 77.6$\pm$ | 1.5
halfcheetah-medium-replay-v2 | 41.3$\pm$ | 0.6 | 41.6$\pm$ | 0.8
hopper-medium-replay-v2 | 84.8$\pm$ | 2.6 | 69.8$\pm$ | 10.1
walker2d-medium-replay-v2 | 66.0$\pm$ | 6.7 | 62.2$\pm$ | 14.4
halfcheetah-medium-expert-v2 | 89.6$\pm$ | 3.0 | 90.6$\pm$ | 2.9
hopper-medium-expert-v2 | 93.2$\pm$ | 20.6 | 89.2$\pm$ | 14.0
walker2d-medium-expert-v2 | 109.3$\pm$ | 0.8 | 106.0$\pm$ | 5.9
Table 10: OTR with Uniform Transport Plan

Table 10 compares the performance of OTR+IQL using the optimal transport plan
and uniform transport plan. We find that for many datasets, using the
suboptimal uniform transport plan is sufficient for reaching good performance.
This indicates that using a reward function based on the similarity of states
from the policy and the expert can be a simple and effective method for reward
labeling. However, note that the uniform transport plan can still underperform
compared to using the optimal transport plan (e.g., hopper-medium-replay-v2).
This shows that the optimal transport formulation enables better and more
consistent performance.

###  A.5 Comparison to PWIL

In this section, we investigate if the online imitation learning algorithm
PWIL (Dadashi et al., 2022) can be used in the offline setting with a change
from using an online RL algorithm to an offline RL algorithm. We ran PWIL with
IQL similar to what we did for OTR in the main paper. We use the PWIL
implementation from Acme (Hoffman et al.,
2022)888https://github.com/deepmind/acme/tree/master/acme/agents/jax/pwil.

Note that although OTR is similar to PWIL in using the Wasserstein distance to
construct RL reward signals, OTR differs from PWIL in the choices of OT
solver, the cost function as well as the approach used for aggregating results
multiple expert demonstrations. Also note that for all experiments in the
paper we consider learning only from expert state instead of state-action
pairs. This is both a more general and challenging setting. It was found in
(Dadashi et al., 2022) that PWIL sometimes perform badly without expert
actions. We ran OTR and PWIL using only expert observations (denoted as OTR-
state and PWIL-state) and OTR and PWIL using state-action pairs (denoted as
OTR-action and PWIL-action). The results are illustrated in table 11. We found
that we are unable to get good results when running PWIL using only expert
state sequences. This is possibly due to difference choices of OT solver and
cost functions. PWIL can perform well when combined with IQL to learn in the
offline setting although sometimes performance is significantly worse compared
to IQL oracle or OTR (e.g., hopper-medium-expert-v2).

K | 10
---|---
method | OTR-state | PWIL-state | OTR-action | PWIL-action
halfcheetah-medium-v2 | 43.1$\pm$ | 0.3 | 1.6$\pm$ | 1.2 | 43.4$\pm$ | 0.3 | 47.5$\pm$ | 0.2
hopper-medium-v2 | 80.0$\pm$ | 5.2 | 2.1$\pm$ | 1.3 | 75.4$\pm$ | 4.6 | 70.4$\pm$ | 4.2
walker2d-medium-v2 | 79.2$\pm$ | 1.3 | 0.9$\pm$ | 1.3 | 79.7$\pm$ | 1.2 | 81.9$\pm$ | 1.0
halfcheetah-medium-replay-v2 | 41.6$\pm$ | 0.3 | -2.3$\pm$ | 0.5 | 41.9$\pm$ | 0.3 | 44.6$\pm$ | 1.1
hopper-medium-replay-v2 | 84.4$\pm$ | 1.8 | 1.4$\pm$ | 1.2 | 85.3$\pm$ | 1.1 | 89.7$\pm$ | 4.9
walker2d-medium-replay-v2 | 71.8$\pm$ | 3.8 | -0.1$\pm$ | 0.2 | 69.1$\pm$ | 4.6 | 72.2$\pm$ | 10.6
halfcheetah-medium-expert-v2 | 87.9$\pm$ | 3.4 | -0.3$\pm$ | 1.5 | 88.3$\pm$ | 5.1 | 88.6$\pm$ | 4.3
hopper-medium-expert-v2 | 96.6$\pm$ | 21.5 | 1.5$\pm$ | 0.6 | 86.6$\pm$ | 22.9 | 32.9$\pm$ | 25.0
walker2d-medium-expert-v2 | 109.6$\pm$ | 0.5 | 1.0$\pm$ | 1.9 | 109.2$\pm$ | 0.5 | 110.2$\pm$ | 0.2
Table 11: Comparison between OTR and PWIL with IQL as offline RL backbone.

In addition, Dadashi et al. (2022) found that PWIL’s performance may
deteriorate when learning from demonstrations consisting of only expert
observations (i.e., no actions are present in the expert demonstrations).

###  A.6 Hyper-parameter Sensitivity

K | 10
---|---
method | OTR ($\alpha=\beta=5$) | OTR ($\alpha=\beta=1$)
halfcheetah-medium-expert-v2 | 87.9$\pm$ | 3.4 | 86.9$\pm$ | 4.0
halfcheetah-medium-replay-v2 | 41.6$\pm$ | 0.3 | 40.4$\pm$ | 1.3
halfcheetah-medium-v2 | 43.1$\pm$ | 0.3 | 42.7$\pm$ | 0.4
hopper-medium-expert-v2 | 96.6$\pm$ | 21.5 | 82.6$\pm$ | 9.9
hopper-medium-replay-v2 | 84.4$\pm$ | 1.8 | 71.2$\pm$ | 15.2
hopper-medium-v2 | 80.0$\pm$ | 5.2 | 75.7$\pm$ | 6.4
walker2d-medium-expert-v2 | 109.6$\pm$ | 0.5 | 106.3$\pm$ | 8.2
walker2d-medium-replay-v2 | 71.8$\pm$ | 3.8 | 63.2$\pm$ | 5.7
walker2d-medium-v2 | 79.2$\pm$ | 1.3 | 77.4$\pm$ | 1.5
Table 12: Effect of $\alpha$ and $\beta$ in the squashing function.

For the main results, the hyper-parameters for the squashing function
($\alpha$ and $\beta$) was chosen to be consistent with those used in (Dadashi
et al., 2022). In this section we compare the differences in the choices of
these hyper-parameters by running OTR with $\alpha=\beta=1$. This reduces to
simply applying an exponential transformation to the optimal transport costs.
The results are illustrated in table 12. We find that OTR still performs well,
demonstrating that it is not sensitive to the choices of these hyper-
parameters.


# Photon correlations in the collective emission of hybrid
gold-(CdSe/CdS/CdZnS) nanocrystal supraparticles

V. Blondot1, D. Gérard1, G. Quibeuf1, C. Arnold1, A. Delteil1, A. Bogicevic2,
T. Pons2, N. Lequeux2, S. Buil1, J-P. Hermier1<EMAIL_ADDRESS>1
Université Paris-Saclay, UVSQ, CNRS, GEMaC, 78000, Versailles, France.
2 Laboratoire de Physique et d’Étude des Matériaux, ESPCI-Paris, PSL Research
University, CNRS UMR 8213, Sorbonne Université, 10 rue Vauquelin, 75005 Paris,
France

###### Abstract

We investigate the photon statistics of the light emitted by single self-
assembled hybrid gold-CdSe/CdS/CdZnS colloidal nanocrystal supraparticles
through the detailed analysis of the intensity autocorrelation function
$g^{(2)}(\tau)$. We first reveal that, despite the large number of
nanocrystals involved in the supraparticle emission, antibunching can be
observed. We then present a model based on non-coherent Förster energy
transfer and Auger recombination that well captures photon antibunching.
Finally, we demonstrate that some supraparticles exhibit a bunching effect at
short time scales corresponding to coherent collective emission.

##  I Introduction

Superradiance is a collective emission phenomenon predicted more than half a
century ago by R. Dicke Dicke54 . Since the 70’s, the concept of superradiance
is at the origin of several multi-faceted works in astrophysics concerning the
amplified diffusion of light Bekenstein73 ; Bekenstein98 ; Brito15 . In the
field of optics, it has also attracted a great attention, motivating
fundamental physics works Rehler71 ; Gross82  as well as studies about the
principles of new laser devices Bonifacio89 .

Due to the inherent homogeneity of their properties, trapped atoms are a
system of choice to experimentally implement the concept of superradiance
Ortiz18 . Regarding condensed-matter emitters, the progress in material
nanostructuration or the use of photonic structures offer the possibility to
tune light matter interaction and to achieve superradiance for a wide range of
nanoemitters such as molecules Luo19 , nanotubes Doria18  or diamond color
centers Vass22 .

In order to promote coherent collective emission, plasmonic nanocavities are
particularly well suited. They allow to confine and tune the electromagnetic
field at subwavelength scales. The plasmonic mode can play a crucial role,
mediating and adjusting the coupling in the near field Pustovit09 ; Pustovit10
; Shlesinger18 . This approach can also be extended to obtain phase locking
between 2D arrays of spasers Dorofeenko13 .

In this paper, we show the collective emission of hybrid plasmonic/colloidal
quantum dot structures. They consist in self-assemblies of core-shell
colloidal CdSe/CdS/CdZnS nanocrystals (NCs) encapsulated into a silica shell
and a gold nanoresonator. The first section of the paper summarizes the
chemical synthesis and the main properties of these golden supraparticles
(GSPs) concerning their photoluminescence decay rate and non coherent Förster
energy transfer (FRET) between single NCs. By performing a detailed time-
resolved analysis of the intensity autocorrelation function $g^{(2)}(\tau)$,
we then show that photon antibunching is observed for several GSPs. This
result is well modeled by a Monte-Carlo simulation taking into account FRET as
well as Auger recombinations and a fraction of non emitting NCs. More
interestingly, we evidence that the light emitted by some GSPs exhibits
bunching at short delays, suggesting coherent collective emission.

##  II Gold-(CdSe/CdS/CdZnS) nanocrystal supraparticles: synthesis and basic
optical properties

A procedure with several steps was implemented to synthesize hybrid gold
CdSe/CdS/CdZnS colloidal NC supraparticles as detailed in Bogicevic2022 .
Briefly, each GSP includes from a few hundred to a few thousand NCs with a
diameter of 7.7 $\pm$ 1 nm (fluorescence centered at 645 nm, FWHM = 30 nm).
Following the approach presented in Blondot20 , the NC aggregate is first
encapsulated in a silica shell with a thickness of 15 nm. It prevents
quenching by the gold nanoshell of the emission of the NCs located at the
surface of the aggregate Lakowicz2005 . After functionalization of the silica
layer, gold seeds are then deposited according to the synthesis reported by
Halas et al. Oldenburg1998 . The synthesis of continuous gold nanoshells with
a thickness of 19 $\pm$ 5 nm is finally achieved (see Figure 1).

Figure 1: Schematic representation of a single GSP (a). Wide field images of
GSPs (b). Typical PL decay rate of an individual GSP (c). It is fitted with a
log-normal distribution with a central decay rate $\tau_{LN}$= 1 ns and a slow
exponential component ($\tau$ = 17 ns) corresponding to the contribution of
non radiative traps.

The gold nanoshell first results in an increase of the PL decay rate through
the well-known Purcell effect that was measured and modeled in detail
Blondot23 . Depending on the GSP diameter, the Purcell factor ranges between 3
and 8, corresponding to PL lifetime between 0.7 ns and 1.9 ns (see Figure
1.c). Close packing the NCs in a compact aggregate also results in the
possibility of Förster resonant energy transfer (FRET) through the well-known
process described by Förster. The small NCs (with the shortest fluorescence
wavelength) act as donors for the larger ones (with a long fluorescence
wavelength). With respect to aggregates without gold nanoshell, we showed that
the gold nanoshell inhibits the contribution of FRET to the total decay rate
of the smallest GSPs by a factor around 3 Blondot23 . In the next section, we
investigate the opportunities opened by the reduction of this incoherent
process in order to achieve coherent interactions between close NCs.

##  III Light emission of the GSPs - photon statistics

###  III.1 Photon counting and confocal microscopy setups

Figure 2: (a) Schematic representation of the $g^{(2)}(\tau)$ time-integration
method. For a given value $T$, we sum the number of coincidences recorded
during a time window $T$ centered around the maximum of each peak Bradac17 .
(b) Zoom of the central peak.

40 µL of the GSP water solution is directly deposited on a glass coverslip
where a TiO2 grid with numbered cells of $65~{}\mu m\times 65~{}\mu m$ was
previously prepared by photolithography. A set of single GSPs is selected by
using an atomic force microscope (AFM) before optical measurements. Finally,
we characterize their form by scanning electron microscopy (SEM). Only
individual GSPs with a shape that is close to a sphere are further considered.

The sample is positioned inside a confocal microscope (Attocube, Attodry 1100,
objective numerical aperture = 0.82) operating at 4 K and equipped with
piezoelectric positioners that enable to excite a selected GSP by a focused
laser beam provided by a pulsed laser diode (Picoquant LDH 520, wavelength =
520 nm, FWHM 160 ps). A dichroic mirror and a fluorescence filter separate the
reflected laser light and the GSP emission, which is collected by a fiber and
sent to a standard Hanbury Brown and Twiss detection setup. The signal of the
two avalanche photodiodes (MPD, time resolution of 50 ps) is recorded by an
acquisition card (Picoquant, PicoHarp 300, time bin of 64 ps) also
synchronized to the pulsed laser diode. From the same data set, the decay of
the luminescence and the histogram of the delays between photons can be
plotted, the latter providing the intensity autocorrelation function
$g^{(2)}(\tau)$.

A pioneer work reporting superradiant emission with quantum dots was based on
the measurement of the enhancement of the PL decay rate Scheibner07 . However,
in the case of most condensed-matter emitters, this approach can lack
reliability due to the possible generation of non-radiative traps during the
fabrication process. Moreover, if cooperative emission is only achieved for a
small fraction of emitters, the signal may be hidden by the standard
fluorescence of the remaining emitters. In contrast, as in many quantum optics
experiments, characterizing the intensity autocorrelation function
$g^{(2)}(\tau)$ appears as a very robust approach to demonstrate collective
emission through the prediction of super-Poissonian and even superthermal
statistics Temnov09 ; Jahnke16 ; Bradac17 . As in Bradac17 , we analyzed the
data by using a time resolved approach consisting in plotting the variations
of the time-integrated function $g^{(2)}(\tau)$ (see figure 2). More
precisely, the area of the peak around zero delay is integrated over a
duration $T$ and normalized by the mean area of the lateral peaks, integrated
over the same duration $T$. This quantity is noted $\tilde{g}^{(2)}_{T}(0)$
and enables to reveal non-Poissonian statistics at short time scale when $T$
decreases.

###  III.2 Results

Thanks to this analysis, different behaviors can be evidenced. The results for
first three GSPs are presented in figure 3. For some GSPs like (a), the photon
statistics is Poissonian whatever the value of $T$. Some GSPs, as shown in (b)
and (c), exhibit a small degree of antibunching that is not expected for non-
interacting emitters. Indeed, if we consider $N$ dipoles emitting
independently, it is well-known that $g^{(2)}(0)$ is equal to the value
$(N-1)/N$, which can be used to develop postselection methods Mangum13 .
However, the emission from clusters of colloidal NCs can exhibit stronger
antibunching due to non-radiative energy transfer Whitcomb14 . Due to Auger
recombinations, energy transfer between two excited NCs results in a first
non-radiative decay channel. It can be compared to the Auger recombination of
a biexcitonic state in a single NC, which is at the very origin of their
single photon emission. Alternatively, blinking or photobleaching can create
another non-radiative recombination channel since an excitation can be
transferred from an ”on” NC to an ”off” one by FRET.

Figure 3: Time-integrated function $\tilde{g}^{(2)}_{T}(0)$ for 3 GSPs showing
a Poissonian statistics (a) or a small amount of antibunching (b,c). The error
bars are calculated by taking into account the finite number of coincidences.

In ref Whitcomb14 , the number of emitters is low ($\leq 4$) so that the
authors could derivate a rate equation model to calculate the amount of
antibunching. In the case of GSPs, such approach is not possible and we
therefore performed Monte-Carlo simulations. We consider an ensemble of $N$
NCs arranged in a cubic lattice with periodic boundary conditions. The
emission energy of each NC is different and is drawn randomly in a set of
equally distributed values. When excited, a NC can transfer its energy by FRET
to one of the 6 closest NCs if its fluorescence wavelength is lower than the
considered adjacent NC. The corresponding rate $k_{FRET}$ is set to the same
value as the radiative one (typical for such self-assembled NCs structures
Blondot20 ; Blondot23 ). In the case of a 6 nm CdS shell thickness, Auger
recombinations at 4 K are partially inhibited due to electron delocalization
into the shell and the radiative quantum efficiency is about 50 % Canneson14 .
Since the NCs used for the present study exhibit a thinner shell (1.6 nm), we
consider that biexcitonic radiative recombinations do not occur. Even if
blinking is nearly suppressed for such kind of emitters at 4 K Canneson14 , we
reported previously that a fraction of the NCs are damaged during the
aggregate synthesis Blondot20 . This is equivalent to the fraction $1-F$ of
NCs in the ”off” state taken into account in Whitcomb14 . More recently, an
in-depth analysis of the first-order coherence of the light emitted by GPSs
allowed us to numerically estimate the fraction $F$ of bright NCs in such a
mesoscopic ensemble Delteil22 . It ranges between 10 % to values exceeding 90
%.

Figure 4: Monte-Carlo simulations of $1-g^{(2)}(0)$ for a cubic crystal of $N$
NCs that can exchange energy by FRET with a rate $k_{FRET}$. The fraction of
emitting NCs is $F$. $k_{rad}$ is the radiative decay rate. The probability to
excite each NC per pulse is 0.3.

Figure 4 shows the variations of $g^{(2)}(0)$ for various parameter values as
a function of the number of NCs. The model predicts an excess amount of
antibunching with respect to the $(N-1)/N$ value corresponding to $N$
independent emitters. As in Whitcomb14 , it increases with $F$ and it reaches
values that are experimentally observed in Figure 3. For some GSPs, the
absence of antibunching is likely to be related to a strong reduction of FRET
processes Blondot23 , large values of $F$ and/or bunching coming from
collective emission.

Before investigating this latter effect, we first evaluate the photon bunching
we would measure from a standard thermal emission characterized by a value
$g^{(2)}(0)=2$. Taking into account the time coherence ($\sim$ 100 fs) deduced
from the linewidth of emission spectrum ($\sim$ 15 nm Blondot23 ) as well as
the time bin in our experiment (64 ps), we calculated the autocorrelation
function $\tilde{g}^{(2)}_{T}(0)$ (see Figure 5). The amount of bunching
$\tilde{g}^{(2)}_{T}(0)-1$ would not exceed a negligible value of 0.03 %.

Figure 5: Calculated time-integrated function $\tilde{g}^{(2)}_{T}(0)$ for a
thermal source with a linewidth of 10 THz (corresponding to the typical
emission linewidth of a GSP).

In contrast with previously shown, up to 20 % bunching is observed for some
GSPs at short values of $T$ (figure 6). These results were only obtained using
an excitation power 10 times lower than previously, in order to reduce as much
as possible processes such as Auger recombinations and promote coherent
interactions between NCs. The probability to excite one NC in the aggregate is
then lower than 10 % (this value is deduced from the evaluation of the setup
collection efficiency, which is about 0.5 %).

When considering the ideal case of $N$ independent and identical dipoles
coupled to a single cavity mode, the cooperativity $C$, which quantifies the
emission into the cavity mode with respect to the other modes, is $N$ times
higher than in a single emitter-in-cavity case Bonifacio76 . This property can
also be seen as an enhancement by a factor $\sqrt{N}$ of the effective
coupling rate between the emitters and the cavity mode. As a result,
collective emission occurs and superradiance is achieved with an emission
decay rate $N$ times higher with respect to the single emitter case Dicke54 .
Let us consider an ensemble of emitters with a fluorescence decay rate
$\gamma$ and coupled to an environment inducing dephasing with a rate
$\gamma^{\ast}$. Qualitatively, when coupled to a cavity with a Purcell factor
$F_{P}$, collective effects can be achieved if $\gamma^{\ast}$ is smaller than
$NF_{P}\gamma$ so that collective emission occurs before the processes at the
origin of the decoherence of the dipoles act Meiser10 .

Figure 6: Time-integrated function $\tilde{g}^{(2)}_{T}(0)$ for 4 GSPs showing
superthermal bunching.

Concerning GSPs, the dephasing rate for a single GSP is typically
$\gamma^{\ast}=$ 10 THz at 4 K Blondot23 ; Delteil22  while $F_{p}\gamma=$ 1
GHz (the mean lifetime is 1 ns, see previous section), leading to the
prediction that collective effects can indeed take place if about $10^{4}$ NCs
are involved. This value can be reached for standard GSPs: taking into account
the structure of the GSPs, the NC diameter, and the volume fraction (66 %)
occupied by the NCs Zaccone22 , one finds that the number of NCs ranges
between 5000 (diameter $\sim$ 220 nm) and 75000 (diameter $\sim$ 400 nm).
While significant, the amount of bunching (20 %) decreases with $T$,
suggesting that only a fraction of NCs are involved in collective emission
processes. A large subset of NCs experience non-collective (and therefore
slower) light emission showing no bunching. As a result,
$\tilde{g}^{(2)}_{T}(0)$ approaches 0 or negative values as $T$ increases. The
absence of bunching for large values of $T$ also demonstrates the relevance of
the time-resolved approach. Not only does it overcome the limitations of
standard methods such as PL decay rate measurements, but it allows us to
discriminate between bunching at short time scales and Poissonian or
antibunching for longer time scales.

##  IV Conclusion

This work focuses on the analysis of the statistical properties of the light
emitted by individual hybrid gold/colloidal NCs structures (GSPs). Using a
specific time-resolved method to analyze the intensity autocorrelation
function $g^{(2)}(\tau)$, we demonstrate that the amount of bunching at short
delays can reach 20 % for a single GSP, showing collective emission. At long
time scales, antibunching is sometimes observed. It fundamentally comes from
energy transfer between adjacent NCs and non radiative recombinations involved
in such NCs aggregates. The realization of hybrid gold-colloidal
nanostructures based on nanoscale emitters with much higher oscillator
strength, such as nanoplatelets Ithurria11 , could open the possibility of
achieving collective emission at room temperature.

## Conflicts of interest

There are no conflicts to declare.

## Acknowledgements

This work was funded by the Agence Nationale de la Recherche in the framework
of the GYN project (Grant No. ANR-17-CE24-0046).

## References

  * (1) R. H. Dicke, ”Coherence in Spontaneous Radiation Processes,” Phys. Rev. 93, 99 (1954)
  * (2) J. Bekenstein, ”Extraction of energy and charge from a black hole,” Phys.Rev. D 7, 949 (1973)
  * (3) J. D. Bekenstein and M. Schiffer, ”The Many faces of superradiance,” Phys.Rev. D 58, 064014 (1998)
  * (4) R. Brito, V. Cardoso, and P. Pani, ”Superradiance, New Frontiers in Black Hole Physics”, 2nd ed., Springer (2020)
  * (5) N. E. Rehler and J. H. Eberly, ”Superradiance,” Phys. Rev. A 3, 1735 (1971)
  * (6) M.Gross and S.Haroche, ”Superradiance: An essay on the theory of collective spontaneous emission,” Phys. Rep 93, 301 (1982)
  * (7) R. Bonifacio, B. W. J. Mc Neil, and P. Pierini, ”Superradiance in the high-gain free-electron laser,” Phys. Rev. A 40, 4467 (1989)
  * (8) L. Ortiz-Gutiérrez, L. F. Muñoz-Martínez, D. F. Barros, J. E. O. Morales, R. S. N. Moreira, N. D. Alves, A. F. G. Tieco, P. L. Saldanha, and D. Felinto ”Experimental Fock-State Superradiance,” Phys. Rev. Lett. 120, 083603 (2018)
  * (9) Y. Luo, G. Chen, Y. Zhang, L. Zhang, Y. Yu, F. Kong, X. Tian, Y. Zhang, C. Shan, Y. Luo, J. Yang, V. Sandoghdar, Z. Dong, and J. G. Hou, ”Electrically Driven Single-Photon Superradiance from Molecular Chains in a Plasmonic Nanocavity,” Phys. Rev. Lett. 122, 233901 (2019)
  * (10) S. Doria, T. S. Sinclair, N. D. Klein, D. I. G. Bennett, C. Chuang, F. S. Freyria, C. P. Steiner, P. Foggi, K. A. Nelson, J. Cao, A. Aspuru-Guzik, S. Lloyd, J. R. Caram, and M. G. Bawendi, ”Photochemical Control of Exciton Superradiance in Light-Harvesting Nanotubes,” ACS Nano 12, 4556 (2018)
  * (11) D. Vass, A. Szenes, B. Bánhelyi, and M. Csete, ”Plasmonically Enhanced Superradiance of Broken-Symmetry Diamond Color Center Arrays Inside Core-Shell Nanoresonators,” Nanomaterials 12, 352 (2022)
  * (12) V. N. Pustovit and T. V. Shahbazyan, ”Cooperative emission of light by an ensemble of dipoles near a metal nanoparticle: The plasmonic Dicke effect,” Phys. Rev. Lett. 102, 077401 (2009)
  * (13) V. N. Pustovit and T. V. Shahbazyan, ”Plasmon-mediated superradiance near metal nanostructures,” Phys. Rev. B 82, 075429 (2010)
  * (14) I. Shlesinger, P. Senellart, L. Lanco, and J.-J. Greffet, ”Tunable bandwidth and nonlinearities in an atom-photon interface with subradiant states,” Phys. Rev. A 98, 013813 (2018)
  * (15) A. V. Dorofeenko, A. A. Zyablovsky, A. P. Vinogradov, E. S. Andrianov, A. A. Pukhov, and A. A. Lisyansky, ”Steady state superradiance of a 2D-spaser array,” Opt. Expr. 21, 14539 (2013)
  * (16) A. Bogicevic, X. Xu, S. Buil, C. Arnold, J.-P. Hermier, T. Pons, and N. Lequeux, ”Synthesis and characterization of colloidal quantum dot superparticles - plasmonic gold nanoshell hybrid nanostructures,” DOI:10.26434/chemrxiv-2022-1m9q5
  * (17) V. Blondot, A. Bogicevic, A. Coste, C. Arnold, S. Buil, X. Quélin, T. Pons, N. Lequeux, J.-P. Hermier, ”Fluorescence properties of self assembled colloidal supraparticles from CdSe/CdS/ZnS nanocrystals,” New J. Phys. 22, 113026 (2020)
  * (18) J. R. Lakowicz, ”Radiative decay engineering 5: metal-enhanced fluorescence and plasmon emission,” Anal. Biochem. 337, 171 (2005)
  * (19) S. J. Oldenburg, R. D. Averitt, S. L. Westcott, and N. J. Halas, ”Nanoengineering of optical resonances,” Chem. Phys. Lett. 288, 243 (1998)
  * (20) V. Blondot, C. Arnold, A. Delteil, D. Gérard, A. Bogicevic, T. Pons, N. Lequeux, J.-P. Hugonin, J.-J. Greffet, S. Buil, J-P. Hermier, ”Fluorescence decay enhancement and FRET inhibition in self-assembled hybrid gold CdSe/CdS/CdZnS colloidal nanocrystals supraparticles,” Opt. Expr. 31, 4454 (2023)
  * (21) M. Scheibner, T. Schmidt, L. Worschech, A. Forchel, G. Bacher, T. Passow, and D. Hommel, ”Superradiance of quantum dots,” Nat. Phys. 3, 106 (2007)
  * (22) V. V. Temnov and U. Woggon, ”Photon statistics in the cooperative spontaneous emission,” Opt. Expr. 17 5774-5782 (2009)
  * (23) F. Jahnke, C. Gies1, M. Aßmann, M. Bayer, H.A.M. Leymann, A. Foerster, J. Wiersig, C. Schneider, M. Kamp and S. Höfling, ”Giant photon bunching, superradiant pulse emission and excitation trapping in quantum-dot nanolasers,” Nat. Comm. 7, 11540 (2016)
  * (24) C. Bradac, M. T. Johnsson, M. van Breugel, B. Q. Baragiola, R. Martin, M. L. Juan, G. K. Brennen, and T. Volz, ”Room-temperature spontaneous superradiance from single diamond nanocrystals,” Nat. Comm. 8 1205 (2017)
  * (25) B. D. Mangum, Y. Ghosh, J. A. Hollingsworth, and H. Htoon, ”Disentangling the effects of clustering and multi-exciton emission in second-order photon correlation experiments,” Opt. Expr. 21, 7419 (2013)
  * (26) D. Canneson, L. Biadala, S. Buil, X. Quélin, C. Javaux, B. Dubertret, J.-P. Hermier, ”Blinking suppression and biexcitonic emission in thick-shell CdSe/CdS nanocrystals at cryogenic temperature,” Phys. Rev. B. 89, 035303 (2014)
  * (27) K. J. Whitcomb, J. Q. Geisenhoff, D. P. Ryan, M. P. Gelfand, and A. Van Orden, ”Photon Antibunching in Small Clusters of CdSe/ZnS Core/Shell Quantum Dots,” J. Phys. Chem. B 119, 9020 (2015)
  * (28) A. Delteil, V. Blondot, S. Buil, and J.-P. Hermier, ”First-order coherence of light emission from inhomogeneously broadened mesoscopic ensembles,” Phys. Rev. B 106, 115302 (2022)
  * (29) R. Bonifacio, and L. Lugiato, ”Cooperative effects and bistability for resonance fluorescence,” Opt. Commun. 19, 172 (1976)
  * (30) D. Meiser and M. J. Holland, ”Steady-state superradiance with alkaline-earth-metal atoms,” Phys. Rev. A 81, 033847 (2010)
  * (31) A. Zaccone, ”Explicit Analytical Solution for Random Close Packing in d=2 and d=3,” Phys. Rev. Lett. 128, 028002 (2022)
  * (32) S. Ithurria, M. D. Tessier, B. Mahler, R. P. S. M. Lobo, B. Dubertret, and Al. L. Efros, ”Colloidal nanoplatelets with two-dimensional electronic structure,” Nat. Mat. 10, 936 (2011)