# Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning

Jianlan Luo Perry Dong Jeffrey Wu Aviral Kumar Xinyang Geng Sergey Levine
University of California, Berkeley

###### Abstract

The offline reinforcement learning (RL) paradigm provides a general recipe to
convert static behavior datasets into policies that can perform better than
the policy that collected the data. While policy constraints, conservatism,
and other methods for mitigating distributional shifts have made offline
reinforcement learning more effective, the continuous action setting often
necessitates various approximations for applying these techniques. Many of
these challenges are greatly alleviated in discrete action settings, where
offline RL constraints and regularizers can often be computed more precisely
or even exactly. In this paper, we propose an adaptive scheme for action
quantization. We use a VQ-VAE to learn state-conditioned action quantization,
avoiding the exponential blowup that comes with naïve discretization of the
action space. We show that several state-of-the-art offline RL methods such as
IQL, CQL, and BRAC improve in performance on benchmarks when combined with our
proposed discretization scheme. We further validate our approach on a set of
challenging long-horizon complex robotic manipulation tasks in the Robomimic
environment, where our discretized offline RL algorithms are able to improve
upon their continuous counterparts by 2-3x. Our project page is at
saqrl.github.io

> Keywords: Offline Reinforcement Learning, Discretization

##  1 Introduction

Offline reinforcement learning (RL) aims to learn policies from static
databases of previously collected experience. As opposed to pure imitation,
offline RL holds the promise to transform logged datasets into policies that
perform _better_ than the behavior policy that collected the dataset by
maximizing task rewards. The key challenge in offline RL is the overestimation
of the value of actions that were not seen in the dataset, which destabilizes
training and leads to policies that perform much worse than their value
function estimates would suggest. To address this issue, a wide variety of
methods have been proposed recently [1, 2, 3, 4, 5]. Typically, these methods
employ some mechanism to stay “close" to behavior policies, such as policy
constraints or value conservatism.

While these methods in principle address overestimation and distributional
shift, in practice implementing these approaches requires various
approximations (depending on the method) that can lead to hyperparameter
sensitivity or performance that is worse than their theoretical formulation
might suggest. This issue is particularly pronounced on “narrow” datasets that
consist of relatively deterministic behavior, such as the demonstration data
that is often used in robotic learning. Our key observation is that these
issues can be mitigated by properly discretizing continuous action spaces, and
then employing discrete action versions of these methods, where many of these
approximations are unnecessary. For example, policy constraints or any
conservatism regularizer that requires expectations over actions can be
computed exactly with discrete actions.

Unfortunately, naïvely discretizing the action space can result in an
exponential blowup in the number of actions, while coarse adaptive
discretization methods can lead to imprecise actions. In this paper, we
propose an adaptive scheme, state-conditioned action quantization (SAQ), for
discretizaing continuous action spaces. We perform state-conditioned action
discretization by utilizing a VQ-VAE model. SAQ is based on a simple insight:
given a particular dataset, there are often only a few in-distribution options
available in each state, corresponding roughly to the “primitives” that are
supported under the data. This allows us to use comparatively very small
discrete action spaces, without suffering from the curse of dimensionality,
while still enjoying the benefits of discrete action spaces and simpler
offline RL algorithm implementations.

The main contribution of this work is a practical approach, SAQ, for learning
quantized action representations for improving continuous-action offline RL
methods on a variety of robotic learning tasks. We present a general method
for learning state-conditioned action discretizations and then apply this
method with three offline RL methods: conservative Q-learning (CQL) [3],
implicit Q-learning (IQL) [4], and behavior regularized actor-critic (BRAC)
[1]. All of these methods require some sort of approximation with continuous
actions, while the discrete version provides for a convenient implementation
that avoids such approximations. We find that the discrete version of each
method implemented with SAQ generally yields improved performance on commonly
used benchmark tasks over each method’s continuous-action counterpart,
particularly on “narrow” datasets (e.g., expert data). We also evaluate these
methods on a set of challenging robotic manipulation tasks from the Robomimic
environment [6], where continuous offline RL methods struggled to get good
performance, and find that our approach outperforms both prior offline RL
methods in this setting by a large margin.

##  2 Related Work

Offline RL. Some offline RL methods use a policy constraint to mitigate
overestimation from distributional shift, constraining the learned policy to
stay close to the data via density modeling or some other divergence measure
[1, 2, 7, 8, 9, 10, 11, 12, 13, 4, 5, 14]. Another line of work directly
regularizes Q values of unseen actions, which we refer as conservative value
function methods [3, 15, 16, 17, 18, 19, 20]. While both types of methods can
be formulated in ways that in principle address distributional shift, in
practice they require some kind of approximation to be applied with continuous
actions, either in estimating the behavior policy, or in computing integrals
or expectations over the action space to formulate pessimism penalties. This
introduces approximation errors and complexity. We show that our approach can
avoid the need for such approximations for three representative methods from
both categories, leading to improved performance.

Discretizing continuous action space for control. Recent work has shown
discretizing continuous actions can yield good performance, and several
discretization strategies have been introduced. To address the issue of
exponential growth of actions in the naïve discretization scheme, one line of
methods assumes independence of action dimensions [21, 22, 23, 24, 25, 26], or
perform discretization in an autoregressive way [27, 28, 29]. Our work is
different in that we perform state-dependant discretization, which adaptively
controls the precision of the discretization scheme. Perhaps the closest to
our work is Dadashi et al. [30], which also learns a state-dependant
discretization. It assumes access to a human demonstration dataset. Given the
current state, it uses a neural network to index a set of plausible continuous
actions and trains the network by comparing its output with the demonstration
data. Our work differs in that we focus on the offline RL setting, where
discretization could enforce conservatism and policy constraints exactly. To
our knowledge, our work is the first to propose adaptive action discretization
with VQ-VAEs for offline RL.

##  3 Preliminaries

The RL problem is formally defined by a Markov decision processes (MDPs)
$\mathcal{M}=(\mathcal{S},\mathcal{A},T,r,\mu_{0},\gamma)$, where
$\mathcal{S}$ and $\mathcal{A}$ denote the state and action spaces, and
$T(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})$, $r(\mathbf{s},\mathbf{a})$
represent the dynamics and reward function, respectively.
$\mu_{0}(\mathbf{s})$ denotes the initial state distribution, and
$\gamma\in(0,1)$ denotes the discount factor. The objective of RL is to learn
a policy that maximizes the return (discounted sum of rewards):
$\max_{\pi}J(\pi):=\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t})\sim\pi}[\sum_{t}\gamma^{t}r(\mathbf{s}_{t},\mathbf{a}_{t})].$
In offline RL, we are provided with an offline dataset
$\mathcal{D}=\\{(\mathbf{s},\mathbf{a},r,\mathbf{s}^{\prime})\\}$ of
transitions collected using a behavior policy ${\pi_{\beta}}$, and our goal is
to find the best possible policy only using the given dataset, without any
additional online data collection. In this paper, we focus on three offline RL
methods, conservative Q-learning (CQL), implicit Q-learning (IQL), and
behavior regularized actor-critic(BRAC); though our approach could likely be
extended to other methods also.

Conservative Q-learning. Naïvely learning a $Q$-value function from the
offline dataset (e.g., via Q-learning or FQI) suffers from OOD actions [31, 2,
32], and the CQL algorithm [3] applies a regularizer $\mathcal{R}(\theta)$ to
prevent querying the target Q-function on unseen actions.
$\mathcal{R}(\theta)$ minimizes the Q-values under the policy
$\pi(\mathbf{a}|\mathbf{s})$, and counterbalances this term by maximizing the
values of the actions in $\mathcal{D}$. Formally, we have the following
objective during Bellman backup:

$\displaystyle\begin{split}\\!\\!\\!\\!\\!\\!\\!\\!{\min_{\theta}~{}\frac{1}{2}\mathop{\mathbb{E}}_{\begin{subarray}{c}\mathbf{s},\mathbf{a},\mathbf{s}^{\prime}\sim\mathcal{D}\\\
\mathbf{a}^{\prime}\sim\pi\end{subarray}}\left[\left(Q_{\theta}(\mathbf{s},\mathbf{a})-r-\gamma\bar{Q}(\mathbf{s}^{\prime},\mathbf{a}^{\prime})\right)^{2}\right]}+\alpha\left(\mathop{\mathbb{E}}_{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim\pi}\left[Q_{\theta}(\mathbf{s},\mathbf{a})\right]-\mathop{\mathbb{E}}_{\mathbf{s},\mathbf{a}\sim\mathcal{D}}\left[Q_{\theta}(\mathbf{s},\mathbf{a})\right]\right),\end{split}$
(1)

where $\bar{Q}$ denotes the target $Q$-function.

Behavior regularized actor-critic. The specific version BRAC-v explicitly
subtracts the divergence
$D(\pi(\cdot|\mathbf{s}^{\prime}),\pi_{\beta}(\cdot|\mathbf{s}^{\prime}))$
from the target value while performing the Bellman update. Additionally, since
the divergence between the learned policy and the behavior policy _at the
current state_ is not a part of the Q-function, BRAC-v also explicitly adds
the divergence value at the current state to the policy update. We instantiate
the version of BRAC that uses the KL-divergence:

$\displaystyle
D_{\mathrm{KL}}(\pi(\cdot|\mathbf{s}),\pi_{\beta}(\cdot|\mathbf{s}))=\mathbb{E}_{\mathbf{a}\sim\pi(\cdot|\mathbf{s})}\left[\log\pi(\mathbf{a}|\mathbf{s})-\log\pi_{\beta}(\mathbf{a}|\mathbf{s})\right].$
(2)

To estimate the second term $\log\pi_{\beta}(\mathbf{a}|\mathbf{s})$, BRAC
trains an additional behavior policy, that we denote as $\hat{\pi}_{\beta}$.
Denoting the policy and the Q-function as $\pi_{\phi}$ and $Q_{\theta}$, the
BRAC-v perform following Bellman update:

$\displaystyle\min_{\theta}~{}~{}\mathbb{E}_{\mathbf{s},\mathbf{a}\sim\mathcal{D}}\left[\left(r(\mathbf{s},\mathbf{a})+\gamma\mathbb{E}_{\mathbf{a}^{\prime}\sim\pi_{\phi}(\cdot|\mathbf{s}^{\prime})}[\bar{Q}_{\theta}(\mathbf{s}^{\prime},\mathbf{a}^{\prime})+\beta\log\hat{\pi}_{\beta}(\mathbf{a}^{\prime}|\mathbf{s}^{\prime})]-Q_{\theta}(\mathbf{s},\mathbf{a})\right)^{2}\right],$
(3)

then extracts a policy by:

$\displaystyle\max_{\phi}\mathop{\mathbb{E}}_{\begin{subarray}{c}\mathbf{s}\sim\mathcal{D}\\\
\mathbf{a}\sim\pi_{\phi}(\cdot|\mathbf{s})\end{subarray}}\left[Q_{\theta}(\mathbf{s},\mathbf{a})+\beta\log\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})-\alpha\log\pi_{\phi}(\mathbf{a}|\mathbf{s})\right]$
(4)

Implicit Q-Learning. Instead of applying policy constraints or critic
regularizations, IQL uses expectile regression to learn a value function to
approximate an expectile $\tau$ over the distribution of actions by optimizing
the value objective

$L_{V}(\psi)=\mathbb{E}_{(\mathbf{s},\mathbf{a})\sim\mathcal{D}}[L_{2}^{\tau}(Q_{\hat{\theta}}(\mathbf{s},\mathbf{a})-V_{\psi}(\mathbf{s}))],$

where $Q_{\hat{\theta}}(\mathbf{s},\mathbf{a})$ is a parameterized target
critic and $L_{2}^{\tau}(u)=|\tau-\mathbb{I}(u<0)|u^{2}$. This value function
is then used to update the Q-function with TD error following the objective
function

$L_{Q}(\theta)=\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})\sim\mathcal{D}}[(r(\mathbf{s},\mathbf{a})+\gamma
V_{\psi}(\mathbf{s}^{\prime})-Q_{\theta}(\mathbf{s},\mathbf{a}))^{2}]$

With the value function $V_{\psi}$ and Q-function
$Q_{\hat{\theta}}(\mathbf{s},\mathbf{a})$, the policy is learned using
advantage weighted regression following the loss below:

$L_{\pi}(\phi)=E_{(\mathbf{s},\mathbf{a})\sim
D}[\exp(\alpha(Q_{\hat{\theta}}(\mathbf{s},\mathbf{a})-V_{\psi}(\mathbf{s})))\log\pi_{\phi}(\mathbf{a}|\mathbf{s})].$

Vector quantized variational autoencoders. Our method uses the VQ-VAE [33] as
part of the discretization process. The VQ-VAE consists of an encoder and a
decoder. The encoder network parameterizes a posterior distribution
$q_{\phi}(z|x)$ over discrete latent variables $z$ given the input data $x$.
The decoder that parameterizes the distribution $p_{\theta}(x|z)$ then
reconstructs the observations from these discrete variables. Specifically, the
encoder first outputs a continuous embedding $e_{\phi}(x)\in R^{D}$.
Discretization is done with a codebook defined on $R^{K\times D}$, composed of
$K$ vectors $e_{j}\in R^{D}$, where $j=1,2,3...K$. The embedding $e_{\phi}(x)$
is compared with all the vectors in the codebook, and the discrete state $z$
is chosen to correspond to the nearest codebook entry. The discrete posterior
is then defined as:

$\displaystyle q(z=k|x)=\begin{cases}1,&\text{if}\
k=\operatorname*{arg\,min}_{j}\lVert e(x)-e_{j}\rVert_{2}\\\
0,&\text{otherwise}\end{cases}$ (5)

Let’s denote $z_{q_{\phi}}(x)$ as the final latent code from the encoder,
i.e., $z_{q_{\phi}}(x)=e_{k}$. The decoder will reconstruct the original input
$x$ based on $z_{q_{\phi}}(x)$. Additionally, a codebook loss and a commitment
loss are added to move codebook vectors and encoder embeddings close to each
other. The overall training objective is as follows:

$\displaystyle L(\theta,\phi,x)=\mathop{\mathbb{E}}_{e\in
e_{j},j=1,2,3...K}\Bigl{[}-\log
p_{\theta}(x|z_{q_{\phi}}(x))+\lVert\text{sg}[e_{\phi}(x)]-e\rVert_{2}^{2}+\lVert
e_{\phi}(x)-\text{sg}[e]\rVert_{2}^{2}\Bigr{]},$ (6)

where sg stands for the stop gradient operator. We can then optimize the
parameters of the encoder and decoder with this objective.

##  4 State Conditioned Action Quantization for Offline RL

The central premise of this paper is that carefully and efficiently
discretizing the action space of a continuous-action RL problem can make it
significantly simpler to implement offline RL methods based on critic or actor
regularization, and that such discretized methods can attain better
performance on especially difficult data distributions, such as narrow
datasets. How should we discretize a continuous action space? Discretizing too
coarsely can make it difficult to perform delicate tasks that require fine
control, and can collapse in-distribution and out-of-distribution actions into
the same bin, making offline RL more difficult. On the other hand, naïvely
using a very fine discretization can be intractable due to the curse of
dimensionality. Our key observation is that we can construct a discretization
with relatively few discrete actions but with minimal loss of resolution if we
employ a learned state-conditioned discretization, which we can accomplish by
means of a VQ-VAE model. In this section, we will describe this approach, and
discuss how it can be combined with simple and effective discrete-action
offline RL methods.

|
---|---
Figure 1: Left: Training SAQ. Right: Discrete policy training/evaluation with
SAQ. SAQ learns a scalar discrete representation of continuous actions using
state-conditioned VQ-VAE. During policy training and evaluation time, we use
the decoder of SAQ to reconstruct the continuous actions from the discrete
policy actions.

###  4.1 State-Conditioned Action Discretization via Scalar-Quantized Auto-
Encoders

Abstractly, we aim to learn a state-conditioned action discretization (SAQ)
scheme that can map a continuous action $\mathbf{a}$ at a given state
$\mathbf{s}$ to a discrete variable $\widehat{a}$ for training, and then given
a choice of the discrete variable $\widehat{a}$ also map it back to the
original action space $\mathcal{A}$ for evaluation.

Given this requirement, a natural choice is to utilize auto-encoders that can
produce state-conditioned discrete codes for a given action. Therefore, our
approach adapts the vector-quantized variational auto-encoder (VQ-VAEs) [33]
framework for performing action discretization, with the difference that
unlike conventional VQ-VAEs that typically learn a multi-dimensional discrete
latent code for an input, we only aim to learn a scalar discrete latent code
for a given input action as this is sufficient for performing downstream RL.

More formally, denoting the parameters of the VQ-VAE encoder with $\phi$ and
the decoder with $\theta$, we wish to learn an encoder $q_{\phi}(\cdot|s,a)$
that produces a one-dimensional discrete latent action, and a decoder to map
the discrete action code back to the original action space,
$p_{\theta}(a|s,\widehat{a})$. Following the training procedure of VQ-VAEs, we
first obtain a latent embedding $z_{q_{\phi}}(s,a)$ from the approximate
posterior $q_{\phi}(\cdot|s,a)$ and then compare it within the codebook, and
obtain the nearest vector $e_{k}$. Next, we run this latent action code
through the decoder to obtain an approximate reconstructed action,
$\tilde{a}\sim p(\cdot|s,e_{k})$, which is further trained to minimize the
reconstruction error against the original action $a$ passed as input to the
encoder. Following the training procedure proposed by Van den Oord et al.
[33], the overall training objective for SAQ is shown in Equation 7, where sg
is the operator of stopping gradient, $\mathcal{D}$ is the offline dataset:

$\displaystyle\min_{\theta,\phi}\mathop{\mathbb{E}}_{\begin{subarray}{c}\mathbf{s},\mathbf{a}\sim\mathcal{D}\\\
e\in e_{j},j=1,2,3...K\end{subarray}}\Bigl{[}-\log
p_{\theta}(\mathbf{a}|z_{q_{\phi}}(\mathbf{s},\mathbf{a}))+\lVert\text{sg}[e_{\phi}(\mathbf{s},\mathbf{a})]-e\rVert_{2}^{2}+\lVert
e_{\phi}(\mathbf{s},\mathbf{a})-\text{sg}[e]\rVert_{2}^{2}\Bigr{]}$ (7)

###  4.2 Offline RL with Discretized Actions

After training VQ-VAE to obtain a discrete action representation, we can
transform the original problem with continuous action into a discrete problem
by learning policies and value functions in the discrete action space.

##### SAQ-CQL.

For CQL, we follow the discrete variant of CQL [3], where we learn the
Q-function with a conservatism penalty and parameterize the policy as
$\pi(\mathbf{a}|\mathbf{s})\propto Q(\mathbf{s},\mathbf{a})$. Specifically, we
employ the maximum entropy variant of CQL and replace the sampled log-
integral-exp with log-sum-exp by summing over all discrete actions. This
allows us to compute the CQL conservatism loss _exactly_ , without having to
rely on the samples from policy to estimate the integral of Q-function. Our
overall objective of SAQ-CQL is therefore

${\min_{\theta}~{}\frac{1}{2}\mathop{\mathbb{E}}_{\begin{subarray}{c}\mathbf{s},\mathbf{a},\mathbf{s}^{\prime}\sim\mathcal{D}\\\
\mathbf{a}^{\prime}\sim\pi\end{subarray}}\left[\left(Q_{\theta}(\mathbf{s},\mathbf{a})-r-\gamma\bar{Q}(\mathbf{s}^{\prime},\mathbf{a}^{\prime})\right)^{2}\right]}+\alpha\mathop{\mathbb{E}}_{\mathbf{s},\mathbf{a}\sim\mathcal{D}}\left[\log\sum_{i}\exp(Q_{\theta}(\mathbf{s},\mathbf{a}_{i}))-Q_{\theta}(\mathbf{s},\mathbf{a})\right].$

We note that in this specific variant of discrete CQL, the conservative
penalty term
${\mathbb{E}}_{\mathbf{s},\mathbf{a}\sim\mathcal{D}}\left[\log\sum_{i}\exp(Q_{\theta}(\mathbf{s},\widehat{\mathbf{a}}_{i}))-Q_{\theta}(\mathbf{s},\widehat{\mathbf{a}})\right]$
is exactly equivalent to the discrete negative log-likelihood behavioral
cloning loss:

$\displaystyle\mathop{\mathbb{E}}_{\mathbf{s},\mathbf{a}\sim\mathcal{D}}\left[\log\sum_{i}\exp(Q_{\theta}(\mathbf{s},\widehat{\mathbf{a}}_{i}))-Q_{\theta}(\mathbf{s},\widehat{\mathbf{a}})\right]$
$\displaystyle=-\mathop{\mathbb{E}}_{\mathbf{s},\mathbf{a}\sim\mathcal{D}}\left[\log\frac{\exp(Q_{\theta}(\mathbf{s},\widehat{\mathbf{a}}))}{\sum_{i}\exp(Q_{\theta}(\mathbf{s},\widehat{\mathbf{a}}_{i}))}\right]$
$\displaystyle=-\mathop{\mathbb{E}}_{\mathbf{s},\mathbf{a}\sim\mathcal{D}}\left[\log\pi_{\theta}(\widehat{\mathbf{a}}|\mathbf{s})\right].$
(8)

##### SAQ-IQL.

For IQL, we follow the original problem formulation in Kostrikov et al. [4],
then derive its closed-form solution in the discrete case. We denote the
advantage as
$A^{\pi}(\mathbf{s},\widehat{\mathbf{a}})=Q^{\pi}(\mathbf{s},\widehat{\mathbf{a}})-V^{\pi}(\mathbf{s})$
for a given policy $\pi$, which can be obtained by the same procedure detailed
in Kostrikov et al. [4]. The policy extraction step in IQL objective is to
find a policy $\pi^{\star}$ so that $A^{\pi}(\mathbf{s},\mathbf{a})$ can be
maximized while staying close to behavior policy $\pi_{\beta}$:

$\displaystyle\pi^{\star}=\operatorname*{arg\,max}\mathbb{E}_{\widehat{\mathbf{a}}\sim\pi(\cdot|\mathbf{s})}\left[A^{\pi}(\mathbf{s},\widehat{\mathbf{a}})\right]\,\,\,\,\text{s.t.}\quad
D_{\mathrm{KL}}(\pi(\cdot|\mathbf{s}),\pi_{\beta}(\cdot|\mathbf{s}))\leq\epsilon.$
(9)

We can solve this constrained optimization in Eq. 9 by using the Lagrangian
method, and the solution is given by:

$\displaystyle\pi^{\star}(\widehat{\mathbf{a}}|\mathbf{s})\propto\exp\left[(\frac{1}{\lambda}A^{\pi}(\mathbf{s},\widehat{\mathbf{a}})+\log\pi_{\beta}(\widehat{\mathbf{a}}|\mathbf{s}))\right],$
(10)

where $\lambda$ is the Lagrangian multiplier, which controls the amount of
constraint deviation. We refer readers to Appendix A for detailed derivation.
In practice, we can obtain $\pi_{\beta}$ by training an additional BC policy
on the behavior dataset.

##### SAQ-BRAC

For SAQ-BRAC, we first learn a categorical discrete-action behavior policy
$\pi_{\beta}(\mathbf{a}|\mathbf{s})$ on the discretized actions by maximizing
the log-likelihood of the offline dataset. We then implement discrete BRAC by
exactly following Eq. 3 and Eq. 4, where we compute the KL-divergence between
the learned policy and behavior policy by simply summing over all discrete
actions.

###  4.3 Diagnosing offline RL performance with constraint enforcement

As conjectured in Sec. 1, we associate inexact constraint enforcement with the
degradation of performance of offline RL methods. To verify this hypothesis,
in this section, we conduct an experiment in a pointmass navigation
environment (maze2d-large from Fu et al. [34]), where the RL agent controls a
point mass to navigate through the maze from a fixed starting point to the
goal point, as shown on the left side of Figure 2. We collect three optimal
demonstration trajectories as the offline dataset and run both continuous CQL
and SAQ-CQL on this domain. We visualize the average return in the middle as
well as the sample estimated (shown in blue) and the exact CQL conservatism
penalty (shown in green) on the right of Figure 2. We observe that the
performance of continuous CQL rapidly degrades with more training steps, with
the estimated conservative penalty $\mathcal{R}(\theta)$ diverging from the
exact value in the period where the performance degrades. The error in
estimation causes the policy training to overly penalize the wrong action
samples, resulting in degraded performance. On the contrary, SAQ-CQL computes
the penalty exactly, which enables the optimizer to smoothly minimize the
penalty, leading to robust policy performance. While it may be that with more
hyperparameter tuning the baseline continuous-action version would perform
better, the results suggest that poor performance in this case corresponds to
the inexact estimate of the CQL regularizer, which is largely mitigated by our
discretization approach.

---
Figure 2: Left: visualization of the maze2d environment. Middle: the average
return of SAQ-CQL and continuous CQL over the course of offline training.
Right: estimated and exact CQL conservatism penalty values. The estimated
conservative penalty for continuous-action CQL diverges significantly from the
true value during training, resulting in rapid performance degradation, while
the SAQ-CQL enjoys stable training by optimizing the exact penalty value. As a
result, the training process (middle) is significantly more stable with SAQ,
which we expect should make the algorithm easier to use, simplifying
checkpoint selection and tuning.

##  5 Experiments

|  |  |
---|---|---|---
Figure 3: D4RL benchmark tasks: locomotion, antmaze, adroit and kitchen.
Figure 4: Robomimic tasks: lift, can, square, tool-hang, transport. Among
these tasks, square features multi-modal behavior, tool-hang requires high-
precision manipulation, transport is particularly challenging because its
long-horizon complex multi-modal nature.

Our experiments compare three continuous-action offline RL methods to their
discretized variants instantiated with SAQ, both on standard offline RL
benchmarks and the Robomimic robotic manipulation environment [6]. We combine
SAQ with: CQL [3], IQL [5], and BRAC [1]. We first evaluate on the D4RL [35]
suite of tasks, and then use the Robomimic tasks [6], which prior work has
found to present a particular challenge for offline RL methods. Appendix B
presents a detailed analysis and ablation study of SAQ to understand how it
affects offline RL training, which we encourage readers to examine for a more
detailed study of the method. We also present a comparison with Aquadem [30]
in Appendix C.

###  5.1 D4RL Benchmark Evaluations

We present the results on the D4RL benchmark suite [34] in Table 5, with
illustrations of the evaluation domains shown in Figure 3. Some of the
benchmark tasks consist of narrow data distributions, while others contain
high-coverage data. For example, the “kitchen-” domains consist of data from
expert human teleoperators controlling a robotic arm to perform a variety of
manipulation skills, which must then be sequenced and recombined by the
algorithm to solve specific tasks. The “-expert” and “-medium-expert” datasets
for the locomotion tasks also contain narrow expert data, possibly in
combination with broad but suboptimal data, with the “-medium-expert” version
presenting a particular challenge in terms of picking out the narrow but high-
performing expert mode from the noisier overall distribution. The “adroit”
dexterous manipulation tasks also contain narrow expert demonstration data
from humans controlling the simulated robotic hand with a data glove. The
narrow datasets are particularly challenging for prior continuous-action
methods, as the narrow behavior policies exacerbate challenges due to
imperfect approximations for policy constraints and conservative regularizers.
This is particularly pronounced for high-dimensional domains, such as the
24-DoF “adroit” robotic hand, where methods that require integrals or
expectations over the action space (such as the regularizer in CQL) must
sample and average over a high-dimensional action space. We see that for each
algorithm (BRAC, IQL, and CQL), the version of each algorithm discretized with
SAQ improves with respect to the average score in each of the domain types
(locomotion, antmaze, adroit, and kitchen). Although in some cases the
improvement is small, in other cases it is very significant, particularly for
narrow-data adroit and kitchen tasks, and the “-medium-expert” versions of the
halfcheetah and hopper task. We hypothesize that SAQ performs well in these
domains because the discretization can capture the individual (narrow) modes,
while the discrete-action RL algorithm can then select from among these modes
to attain the best performance.

Task | BRAC | SAQ-BRAC | IQL | SAQ-IQL | CQL | SAQ-CQL | BC | SAQ-BC
---|---|---|---|---|---|---|---|---
locomotion avg | 64.16 | 66.98 | 75.14 | 76.67 | 75.6 | 77.35 | 38.86 | 60.77
antmaze avg | 18.84 | 29.5 | 55.34 | 55.92 | 55.15 | 56.87 | 0 | 0
adroit avg | 23.86 | 40.09 | 20.32 | 22.82 | 10.58 | 21.37 | 12.35 | 21
kitchen avg | 15.78 | 36.11 | 50.83 | 58.61 | 48.67 | 65.89 | 17.22 | 57
Table 1: Averaged normalized scores across locomotion, Adroit, AntMaze, and
kitchen domains from D4RL. The version of each algorithm discretized with SAQ
generally improves over the average score of the original algorithm in each
class of domains, with particularly pronounced improvements on narrow dataset
domains such as adroit and kitchen. The full results are deferred to Appendix
C

While SAQ can lead to significant improvements over the continuous-action
counterpart of each method, the main benefit of SAQ is not in raw performance,
but in terms of the simplification of the downstream RL problem. Our
experiments provide an “apples to apples” comparison for each RL method,
comparing its discrete (SAQ) version to its original continuous formulation,
but of course, other offline RL methods might perform better on some tasks,
and we do not aim to show that SAQ achieves state-of-the-art performance over
all possible offline RL techniques. However, we expect that the simple
discrete-action RL problem that is presented via our proposed discretization
will in the long run make it easier to scale offline RL to harder and more
complex problem domains, and provide a valuable tool to the RL practitioner.
RL methods tend to be complex and difficult to tune, so any improvement that
simplifies the RL part of the problem is likely to improve practical utility.

###  5.2 Robomimic Evaluation

Robomimic [6] is a set of environments and demonstration datasets that require
controlling a 7-DoF robot arm to perform a variety of manipulation tasks.
Prior work [6] reported that these environments are particularly challenging
for offline RL algorithms, with the best-reported results obtained by simpler
imitation learning methods. Using the “proficient human” (PH) datasets, which
each consist of 200 successful trajectories with a binary reward, we trained
policies with the continuous version of IQL, CQL, and BC, as well as discrete-
action policies using the discretization from SAQ. The results, presented in
Table 2, show that SAQ indeed improves continuous offline RL by large margins
on all the tasks considered. It’s worthwhile to mention that our continuous BC
results don’t exactly match the original paper [6], as the authors point out,
they adopt a Gaussian Mixture Model (GMM) for the policy class and extensively
optimize parameters then perform checkpoint selection, whereas we directly
train unimodal BC policies.

Task |  IQL |  SAQ-IQL |  CQL |  Robomimic CQL |  SAQ-CQL |  BC |  Robomimic BC |  SAQ-BC
---|---|---|---|---|---|---|---|---
lift |  58 |  90 |  64.2 |  92.7 |  90.8 |  59.47 |  100 |  90.13
can |  33.73 |  68 |  19.6 |  38 |  71.2 |  31.73 |  95.3 |  66.4
square |  26.93 |  46.67 |  0 |  5.3 |  44.27 |  19.33 |  78.7 |  45.33
tool-hang |  2.67 |  28 |  0 |  0 |  3.87 |  1.87 |  17.3 |  3.47
transport |  0 |  2 |  0 |  0 |  3.47 |  0.27 |  29.3 |  3.2
average |  24.27 |  46.93 |  16.76 |  27.2 |  42.72 |  22.53 |  64.12 |  41.71
Table 2: Average success rates on Robomimic tasks using the Proficient Human
dataset for each task.

While the D4RL results suggest that discretization with SAQ can consistently
improve the performance of each offline RL algorithm, these results further
suggest that domains that are especially challenging for offline RL, such as
narrow demonstration datasets, are particularly amenable for SAQ, where it can
enable offline RL methods that were previously outperformed by simple
imitation learning to attain significantly better results. This suggests that
SAQ can be a particularly effective tool in robotic learning, where narrow
demonstration datasets might be commonplace.

##  6 Discussion, Limitations, and Future Work

We presented a method for state-conditioned action quantization to improve
continuous offline RL algorithms. Our approach allows offline RL methods to
enforce policy constraints or value conservatism more exactly as compared to
their continuous counterparts. This is particularly relevant and important in
the robotic learning setting where we usually assume narrow datasets of expert
demonstrations where function approximation errors get even more exaggerated.
However our approach does have a number of limitations. First, we require
sufficient state-action coverage to be able to perform state-conditioned
action quantization. This is a common assumption in offline RL and is true in
many curated datasets; however, it might be challenging to obtain such
datasets in real-world robotic settings. That said, our approach is able to
achieve impressive performance in the Robomimic environment which largely
composed of real human teleoperation data. Second, it is not clear the best
way to adopt our method in the online finetuning setting, where new data might
invalidate the learned discretization. Adaptively adjusting the discretization
during online training could be a valuable topic to explore in future work.

#### Acknowledgments

This research was partly supported through the Office of Naval Research
through N00014-21-1-2838 and N00014-20-1-2383. We acknowledge computing
support from the Berkeley Research Computing (BRC) program and the NSF
Cloudbank program.

## References

  * Wu et al. [2019] Y. Wu, G. Tucker, and O. Nachum.  Behavior regularized offline reinforcement learning.  _arXiv preprint arXiv:1911.11361_ , 2019.
  * Kumar et al. [2019] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine.  Stabilizing off-policy q-learning via bootstrapping error reduction.  In _Advances in Neural Information Processing Systems_ , pages 11761–11771, 2019.
  * Kumar et al. [2020] A. Kumar, A. Zhou, G. Tucker, and S. Levine.  Conservative q-learning for offline reinforcement learning.  _arXiv preprint arXiv:2006.04779_ , 2020.
  * Kostrikov et al. [2021a] I. Kostrikov, A. Nair, and S. Levine.  Offline reinforcement learning with implicit q-learning.  _arXiv preprint arXiv:2110.06169_ , 2021a.
  * Kostrikov et al. [2021b] I. Kostrikov, J. Tompson, R. Fergus, and O. Nachum.  Offline reinforcement learning with fisher divergence critic regularization.  _arXiv preprint arXiv:2103.08050_ , 2021b.
  * Mandlekar et al. [2021] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, F.-F. Li, S. Savarese, Y. Zhu, and R. Martín-Martín.  What matters in learning from offline human demonstrations for robot manipulation.  In _5th Annual Conference on Robot Learning_ , 2021.  URL https://openreview.net/forum?id=JrsfBJtDFdI.
  * Fujimoto et al. [2019] S. Fujimoto, D. Meger, and D. Precup.  Off-policy deep reinforcement learning without exploration.  In _Proceedings of the 36th International Conference on Machine Learning_ , 2019.
  * Jaques et al. [2019] N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard.  Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.  _arXiv preprint arXiv:1907.00456_ , 2019.
  * Peng et al. [2019] X. B. Peng, A. Kumar, G. Zhang, and S. Levine.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  _arXiv preprint arXiv:1910.00177_ , 2019.
  * Nair et al. [2020] A. Nair, M. Dalal, A. Gupta, and S. Levine.  Accelerating online reinforcement learning with offline datasets.  _arXiv preprint arXiv:2006.09359_ , 2020.
  * Wang et al. [2020] Z. Wang, A. Novikov, K. Żołna, J. T. Springenberg, S. Reed, B. Shahriari, N. Siegel, J. Merel, C. Gulcehre, N. Heess, et al.  Critic regularized regression.  _arXiv preprint arXiv:2006.15134_ , 2020.
  * Fujimoto and Gu [2021] S. Fujimoto and S. S. Gu.  A minimalist approach to offline reinforcement learning.  _arXiv preprint arXiv:2106.06860_ , 2021.
  * Peters and Schaal [2007] J. Peters and S. Schaal.  Reinforcement learning by reward-weighted regression for operational space control.  In _International Conference on Machine Learning_ , 2007.
  * Siegel et al. [2020] N. Y. Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, and M. Riedmiller.  Keep doing what worked: Behavioral modelling priors for offline reinforcement learning.  _arXiv preprint arXiv:2002.08396_ , 2020.
  * Xie et al. [2021] T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal.  Bellman-consistent pessimism for offline reinforcement learning.  _Advances in neural information processing systems_ , 34, 2021.
  * Nachum et al. [2019] O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans.  Algaedice: Policy gradient from arbitrary experience.  _arXiv preprint arXiv:1912.02074_ , 2019.
  * Yu et al. [2020] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma.  Mopo: Model-based offline policy optimization.  _arXiv preprint arXiv:2005.13239_ , 2020.
  * Yu et al. [2021] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn.  Combo: Conservative offline model-based policy optimization.  _arXiv preprint arXiv:2102.08363_ , 2021.
  * Jin et al. [2020] Y. Jin, Z. Yang, and Z. Wang.  Is pessimism provably efficient for offline rl?  _arXiv preprint arXiv:2012.15085_ , 2020.
  * Rezaeifar et al. [2021] S. Rezaeifar, R. Dadashi, N. Vieillard, L. Hussenot, O. Bachem, O. Pietquin, and M. Geist.  Offline reinforcement learning as anti-exploration.  _arXiv preprint arXiv:2106.06431_ , 2021.
  * Tavakoli et al. [2018] A. Tavakoli, F. Pardo, and P. Kormushev.  Action branching architectures for deep reinforcement learning.  2018\.
  * Vieillard et al. [2022] N. Vieillard, M. Andrychowicz, A. Raichuk, O. Pietquin, and M. Geist.  Implicitly regularized rl with implicit q-values.  In _Proceedings of The 25th International Conference on Artificial Intelligence and Statistics_ , 2022.
  * Tang and Agrawal [2019] Y. Tang and S. Agrawal.  Discretizing continuous action space for on-policy optimization.  2019\.
  * Chebotar et al. [2023] Y. Chebotar, Q. Vuong, K. Hausman, F. Xia, Y. Lu, A. Irpan, A. Kumar, T. Yu, A. Herzog, K. Pertsch, K. Gopalakrishnan, J. Ibarz, O. Nachum, S. A. Sontakke, G. Salazar, H. T. Tran, J. Peralta, C. Tan, D. Manjunath, J. Singh, B. Zitkovich, T. Jackson, K. Rao, C. Finn, and S. Levine.  Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions.  In _7th Annual Conference on Robot Learning_ , 2023.  URL https://openreview.net/forum?id=0I3su3mkuL.
  * Shafiullah et al. [2022] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto.  Behavior transformers: Cloning $k$ modes with one stone.  In _Thirty-Sixth Conference on Neural Information Processing Systems_ , 2022.
  * Seyde et al. [2022] T. Seyde, P. Werner, W. Schwarting, I. Gilitschenski, M. Riedmiller, D. Rus, and M. Wulfmeier.  Solving continuous control via q-learning.  arXiv, 2022.
  * Metz et al. [2017] L. Metz, J. Ibarz, N. Jaitly, and J. Davidson.  Discrete sequential prediction of continuous actions for deep rl.  arXiv, 2017.
  * Tessler et al. [2019] C. Tessler, G. Tennenholtz, and S. Mannor.  Distributional policy optimization: An alternative approach for continuous control.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Tavakoli et al. [2021] A. Tavakoli, M. Fatemi, and P. Kormushev.  Learning to represent action values as a hypergraph on the action vertices.  In _International Conference on Learning Representations_ , 2021.
  * Dadashi et al. [2022] R. Dadashi, L. Hussenot, D. Vincent, S. Girgin, A. Raichuk, M. Geist, and O. Pietquin.  Continuous control with action quantization from demonstrations.  In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 4537–4557, 17–23 Jul 2022.
  * Fujimoto et al. [2018] S. Fujimoto, D. Meger, and D. Precup.  Off-policy deep reinforcement learning without exploration.  _arXiv preprint arXiv:1812.02900_ , 2018.
  * Levine et al. [2020] S. Levine, A. Kumar, G. Tucker, and J. Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Van den Oord et al. [2017] A. Van den Oord, O. Vinyals, and k. kavukcuoglu.  Neural discrete representation learning.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Fu et al. [2020a] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020a.
  * Fu et al. [2020b] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  In _arXiv_ , 2020b.  URL https://arxiv.org/pdf/2004.07219.

## Appendix

##  Appendix A Detailed SAQ-IQL derivation

In this section, we describe in detail the derivation for the SAQ-IQL
algorithm. We start with the original optimization objective from IQL that
employs an explicit policy constraint w.r.t. to a behavior policy

$\begin{split}\pi^{*}&=\operatorname*{arg\,max}_{\pi\in\Pi}\mathbb{E}_{\mathbf{a}\sim\pi(\cdot|\mathbf{s})}[A^{\pi}(\mathbf{s},\mathbf{a})]\\\
\text{s.t.
}&D_{KL}(\pi(\cdot,\mathbf{s})||\pi_{\beta}(\cdot|\mathbf{s}))\leq\epsilon\\\
&\sum_{\mathbf{a}}\pi(\mathbf{a}|\mathbf{s})=1\end{split}$ (11)

We can write down the Lagrangian of 11 and expand it out:

$L(\pi,\lambda,\alpha)=\mathbb{E}_{\mathbf{a}\sim\pi(\cdot|\mathbf{s})}[A^{\pi}(\mathbf{s},\mathbf{a})]+\lambda(\epsilon-
D_{KL}(\pi(\cdot|\mathbf{s})||\pi_{\beta}(\cdot|\mathbf{s})))+\alpha(1-\sum_{\mathbf{a}}\pi(\mathbf{a}|\mathbf{s}))$
$L(\pi,\lambda,\alpha)=\sum_{a\sim\pi(\cdot|s)}(\pi(\mathbf{a}|\mathbf{s})A^{\pi}(\mathbf{s},\mathbf{a}))+\lambda(\epsilon-\sum_{\mathbf{a}\sim\pi(\cdot|\mathbf{s})}(\pi(\mathbf{a}|\mathbf{s})\log(\frac{\pi(\mathbf{a}|\mathbf{s})}{\pi_{\beta}(\mathbf{a}|\mathbf{s})})))+\alpha(1-\sum_{\mathbf{a}}\pi(\mathbf{a}|\mathbf{s}))$

Differentiating $L(\pi,\lambda,\alpha)$ with respect to
$\pi(\mathbf{a}|\mathbf{s})$ results in

$\frac{\partial
L(\pi,\lambda,\alpha)}{\partial\pi}=A^{\pi}(\mathbf{s},\mathbf{a})-\lambda(\log(\pi(\mathbf{a}|\mathbf{s}))-\log(\pi_{\beta}(\mathbf{a}|\mathbf{s}))+1)-\alpha$

We divide the differentiated Lagrangian by the constant $\lambda$

$\frac{\partial
L(\pi,\lambda,\alpha)}{\partial\pi}\cdot\frac{1}{\lambda}=\frac{A^{\pi}(\mathbf{s},\mathbf{a})}{\lambda}+\log(\pi_{\beta}(\mathbf{a}|\mathbf{s}))-\log(\pi(\mathbf{a}|\mathbf{s}))-1-\frac{\alpha}{\lambda}$

and set the resulting expression to 0 to arrive at
$\pi^{*}(\mathbf{a}|\mathbf{s})$,

$\pi^{*}(\mathbf{a}|\mathbf{s})=\exp[\frac{A^{\pi}(\mathbf{s},\mathbf{a})}{\lambda}+\log(\pi_{\beta}(\mathbf{a}|\mathbf{s}))-1-\frac{\alpha}{\lambda}]$
$\pi^{*}(\mathbf{a}|\mathbf{s})=\frac{1}{z(\mathbf{s})}\exp[\frac{A^{\pi}(\mathbf{s},\mathbf{a})}{\lambda}+\log(\pi_{\beta}(\mathbf{a}|\mathbf{s}))]$

where $z(\mathbf{s})$ is a normalizing term.

##  Appendix B Ablation Studies for SAQ

Task |  No State |  State |  No State |  State |  No State |  State
---|---|---|---|---|---|---
|  SAQ- |  SAQ- |  SAQ- |  SAQ- |  SAQ- |  SAQ-
|  CQL |  CQL |  IQL |  IQL |  BRAC |  BRAC
halfcheetah-medium-replay-v2 |  1.56 |  47.07 |  1.56 |  36.2 |  -1.6 |  40.25
hopper-medium-replay-v2 |  15.24 |  94.73 |  11.74 |  59.43 |  21.56 |  68.87
walker2d-medium-replay-v2 |  4.67 |  81.72 |  6.89 |  45.64 |  -0.25 |  53.52
average |  7.16 |  74.51 |  6.73 |  47.09 |  6.57 |  54.21

Table 3: Comparing the performance of state-conditioned action discretization
against unconditioned action discretization with CQL. The state-conditioned
discretization scheme significantly outperforms the unconditioned one since
unconditioned action discretization cannot compress the action space into few
number of bins.

Comparing discretization methods. To understand the importance of the state-
conditioned discretization method, we compare it against a naive
discretization method where the VQ-VAE discretizes the actions without
conditioning on the states and present the results in Table 3. We see that the
state-conditioning allows is indeed highly important in compressing the action
space into a small number of bins, resulting in much higher performance than a
state-agnostic discretization scheme.

Codebook size robustness. One key design choice we make in this paper is the
use of a VQ-VAE, one natural question is then our method’s robustness against
the codebook size in the VQ-VAE; since that can be crucial in determining the
quality of the performed discretization through it. Towards this end, we
empirically experiment with varying codebook sizes across all three
algorithms. We present the results in Table 4, and we found that our method’s
performance is consistent across codebook sizes; which further resonates with
the practical utility of adopting our method.

Codebook Size | 16 | 32 | 64 | 128
---|---|---|---|---
SAQ-CQL | 108.7 | 111.6 | 110.8 | 103.2
SAQ-IQL | 106.9 | 104.8 | 104.2 | 94.42
SAQ-BRAC | 106.5 | 108.3 | 105.7 | 107

Table 4: Comparing the performance of SAQ-IQL, SAQ-CQL, and SAQ-BRAC on
hopper-expert-v2 while varying the codebook size. It can be observed that the
discretized algorithms are invariant to codebook size changes.

Figure 5: Increasing policy constraint levels on hopper-expert-v2 environment
from Gym locomotion.

Controlling policy constraint levels. As stated in Sec.4.1, one key premise of
SAQ is that we can enforce policy constraint or value conservatism exactly;
which associates with the practical performance of offline RL methods. To
further verify this hypothesis empirically; we pick one task from the Gym
locomotion suite and vary the weight coefficients for policy constraint or
value conservatism to observe the resulting performance. We present our
results in Fig. 5, we can see that small constraint enforcement leads to poor
performance initially; then the performance ramps up when we increase the
coefficients; finally converges with sufficient large coefficients. This
observation confirms our conjecture in the paper.

##  Appendix C Additional Experiment Results

Task |  BRAC |  SAQ-BRAC |  IQL |  Aquadem-IQL |  SAQ-IQL |  CQL |  Aquadem-CQL |  SAQ-CQL |  BC |  Aquadem-BC |  SAQ-BC
---|---|---|---|---|---|---|---|---|---|---|---
halfcheetah-expert-v2 |  67.99 |  64 |  94.78 |  92.73 |  90.3 |  43.9 |  92.66 |  92.65 |  91.53 |  92.69 |  108.4
halfcheetah-medium-expert-v2 |  73.07 |  90.31 |  88.06 |  80.75 |  89.05 |  50.32 |  88.3 |  91.69 |  48.81 |  74.17 |  57.65
halfcheetah-medium-replay-v2 |  -3.3 |  35.28 |  44.24 |  35.12 |  36.2 |  47.07 |  40.5 |  40.25 |  35.16 |  35.62 |  3.76
halfcheetah-medium-v2 |  47.08 |  43 |  47.3 |  42.39 |  42.52 |  48.56 |  44.5 |  43.89 |  42.32 |  41.93 |  47.02
hopper-expert-v2 |  67.34 |  110 |  108.8 |  109 |  100.3 |  100.5 |  110.3 |  109.9 |  98.8 |  110.1 |  92.63
hopper-medium-expert-v2 |  50.74 |  98.75 |  32.85 |  54.57 |  81.56 |  67.08 |  86.7 |  98.47 |  44.48 |  57.9 |  58.94
hopper-medium-replay-v2 |  36.81 |  32.54 |  61.75 |  38.05 |  59.43 |  94.73 |  90.3 |  68.87 |  17.96 |  20.94 |  32.93
hopper-medium-v2 |  57.06 |  54.16 |  54.3 |  49.2 |  50.53 |  70.32 |  58.5 |  38.25 |  50.31 |  52.98 |  42.47
walker2d-expert-v2 |  108.4 |  107.5 |  110.12 |  108.5 |  107.6 |  109.2 |  108.2 |  107.3 |  108.3 |  108.3 |  104.5
walker2d-medium-expert-v2 |  109 |  102.9 |  109.56 |  108 |  100.3 |  110.7 |  108.1 |  108.6 |  91.79 |  83.55 |  103.2
walker2d-medium-replay-v2 |  73.72 |  0.4 |  68.71 |  32.66 |  45.64 |  81.72 |  80.8 |  53.52 |  12.98 |  27.86 |  15.27
walker2d-medium-v2 |  81.94 |  64.8 |  81.25 |  68.06 |  68 |  83.11 |  82.1 |  74.77 |  70.18 |  27.86 |  62.52
locomotion average |  64.16 |  66.98 |  75.14 |  71.01 |  76.67 |  75.6 |  82.58 |  77.35 |  59.39 |  61.16 |  60.77
antmaze-medium-diverse-v2 |  26 |  47.6 |  76.67 |  40 |  68.33 |  72.75 |  22.67 |  75.47 |  0 |  0 |  0
antmaze-medium-play-v2 |  48.67 |  56.93 |  78.67 |  46 |  74.33 |  67.04 |  30.67 |  68.67 |  0 |  1.3 |  0
antmaze-large-diverse-v2 |  0.66 |  9.73 |  31.67 |  33.67 |  41 |  35.62 |  22.67 |  36 |  0 |  0 |  0
antmaze-large-play-v2 |  0 |  3.73 |  34.33 |  14.33 |  40 |  45.18 |  25.33 |  47.33 |  0 |  0 |  0
antmaze average |  18.84 |  29.5 |  55.34 |  33.5 |  55.92 |  55.15 |  25.34 |  56.87 |  0 |  0.33 |  0
door-human-v0 |  $-1.01$ |  35.42 |  1.79 |  2.15 |  9.26 |  0.84 |  6.09 |  2.12 |  3.29 |  3.24 |  9.28
hammer-human-v0 |  $-1.42$ |  20.52 |  1.41 |  1.56 |  1.57 |  0.27 |  2.64 |  0.6 |  0.8 |  2.09 |  1.38
pen-human-v0 |  98.15 |  98.41 |  69.69 |  73.91 |  80.25 |  41.24 |  85.66 |  82.73 |  45.28 |  72.12 |  73.3
relocate-human-v0 |  $-0.28$ |  6 |  8.38 |  0.55 |  0.2 |  $-0.05$ |  0.28 |  0.02 |  0.04 |  0.48 |  0.02
adroit average |  23.86 |  40.09 |  20.32 |  19.54 |  22.82 |  10.58 |  23.67 |  21.37 |  12.35 |  19.48 |  21
kitchen-mixed-v0 |  10.33 |  53.33 |  48.92 |  56.94 |  52.92 |  62 |  0 |  57.67 |  37.9 |  58.89 |  34
kitchen-complete-v0 |  31.67 |  10 |  66 |  70.92 |  76.76 |  14 |  21.5 |  47.67 |  39.13 |  68 |  90.33
kitchen-partial-v0 |  5.33 |  45 |  37.58 |  44.83 |  46.25 |  70 |  50 |  92.33 |  37.33 |  43.56 |  46.67
kitchen average |  15.78 |  36.11 |  50.83 |  57.56 |  58.61 |  48.67 |  23.83 |  65.89 |  38.12 |  56.82 |  57

Table 5: Comparison of our method and Aquadem on various D4RL tasks. SAQ in
general improves over its continuous counterpart, especially in the narrow
dataset setting; also outperforms Aquadem in most tasks.


# SoccerNet-Echoes: A Soccer Game Audio Commentary Dataset

Sushant Gautam1,2 Mehdi Houshmand Sarkhoosh2,3 Jan Held5 Cise Midoglu1,3
Anthony Cioppa5,6 Silvio Giancola6 Vajira Thambawita1
Michael A. Riegler1 Pål Halvorsen1,2,3 Mubarak Shah4
1SimulaMet, Norway 2OsloMet, Norway 3Forzasys, Norway
4University of Central Florida, USA 5University of Liège, Belgium 6KAUST,
Saudi Arabia
https://github.com/SoccerNet/sn-echoes

###### Abstract

The application of Automatic Speech Recognition (ASR) technology in soccer
offers numerous opportunities for sports analytics. Specifically, extracting
audio commentaries with ASR provides valuable insights into the events of the
game, and opens the door to several downstream applications such as automatic
highlight generation. This paper presents SoccerNet-Echoes, an augmentation of
the SoccerNet dataset with automatically generated transcriptions of audio
commentaries from soccer game broadcasts, enhancing video content with rich
layers of textual information derived from the game audio using ASR. These
textual commentaries, generated using the _Whisper_ model and translated with
_Google Translate_ , extend the usefulness of the SoccerNet dataset in diverse
applications such as enhanced action spotting, automatic caption generation,
and game summarization. By incorporating textual data alongside visual and
auditory content, SoccerNet-Echoes aims to serve as a comprehensive resource
for the development of algorithms specialized in capturing the dynamics of
soccer games. We detail the methods involved in the curation of this dataset
and the integration of ASR. We also highlight the implications of a multimodal
approach in sports analytics, and how the enriched dataset can support diverse
applications, thus broadening the scope of research and development in the
field of sports analytics.

##  1 Introduction

Figure 1: The pipeline for generating a multilingual commentary dataset from
soccer game videos, incorporating different Whisper versions, language
detection and translation.

Sports analytics has progressively embraced technological innovations, to
enrich the analysis and understanding of complex sports events. Historically,
analysis of sports video footage relied primarily on visual cues: action
spotting, player recognition and player tracking were based solely on videos
and video frames as images. However, with the advancement in multimodal data
integration, the scope of sports analytics has broadened, encompassing audio
and textual data alongside traditional visual input. This integration
facilitates a more holistic view of sports events, capturing the nuances of
game dynamics that visual data alone might miss.

The SoccerNet dataset [12, 7], initially designed to support video-based
analyses such as action spotting, player tracking [5], camera calibration [4],
dense video captioning [24], foul recognition [14, 15], and game state
reconstruction [31], has emerged as a key resource in sports video analytics.
Yet, the rapidly evolving field of sports analytics demands ever more
sophisticated tools that can process and interpret the complex interplay of
multiple data modalities, including video, audio, and text. In response to
this need, we augment the SoccerNet dataset by incorporating state-of-the-art
ASR technology to enrich the dataset with textual information derived from
live game commentaries. This multimodal approach aims to refine the accuracy
of existing tasks such as action spotting and dense video captioning, as well
as extends the dataset’s usefulness in more complex applications such as
sentiment analysis or tactical assessment.

This paper details our augmentation of the SoccerNet dataset to include audio
commentary transcriptions, generated with OpenAI’s Whisper [25] tool, and
translated with Google Translate [13]. The resulting SoccerNet-Echoes dataset
serves as a comprehensive toolkit for researchers and practitioners, allowing
enhanced interpretation and narrative generation from sports footage. Through
the integration of audio commentaries in textual format, SoccerNet-Echoes
facilitates a deeper understanding of the contextual aspects of soccer games,
thereby enhancing both the analytical capabilities and the viewer’s
experience.

Contributions. Our contributions are threefold: (i) We augment the SoccerNet
dataset with automatically transcribed and translated audio commentaries from
soccer game broadcasts in multiple languages, and present this new dataset
(SoccerNet-Echoes) as an open-access resource. (ii) We evaluate SoccerNet-
Echoes using human-verified ASR from literature and perform quantitative
analysis. (iii) We explore the potential applications of SoccerNet-Echoes in
various research areas, and highlight the broader implications of adopting a
multimodal approach for sports research in general.

##  2 Background and Related Work

The integration of ASR technology has significantly advanced the analysis and
processing of video content, impacting sports research [22, 20, 21, 3].
Initially, video understanding focused primarily on computer vision tasks,
such as action recognition and event spotting [32, 33, 29, 2]. Advancements in
dense video captioning [18] began to explore the generation of captions for
temporally localized activities within untrimmed videos, representing a
significant departure from the traditional approach of generating a single
caption for short clips. Further evolution in the field saw the incorporation
of audio and speech modalities alongside visual data, enhancing video
captioning capabilities.

###  2.1 Early Applications: Action Recognition and Event Detection

Before the integration of ASR, the primary focus in sports video analysis was
on visual cues for game action recognition and event detection (spotting) [12,
7]. Researchers began to explore the use of multimodal sports data, combining
game audio and video streams to improve the accuracy of action spotting in
soccer videos [1, 34]. These studies marked an early recognition of the
potential benefits of incorporating audio data alongside traditional video
processing, setting the stage for the subsequent integration of ASR
technologies in sports game analysis [9].

###  2.2 Transition to Speech and Language Processing

As the potential of multimodal analysis became evident, subsequent studies
began to integrate more sophisticated speech and language processing
technologies [10, 11]. Gao et al. [8] utilized ASR to segment and categorize
commentary from soccer videos, thereby enhancing key moment extraction and
highlight generation. This application of ASR for segmenting commentator
speech represented a shift towards leveraging linguistic information to
complement visual data. Gautam et al. used captions, text and commentaries
extracted from ASR for game summarization [10], and used audio intensity to
capture the field excitement [11].

###  2.3 Multimodal Dense Video Captioning

The application of textual modality in sports video analysis reached a new
level with the introduction of dense video captioning tasks, as exemplified by
the SoccerNet Challenge [6]. The SoccerNet-Caption dataset [23] leverages
rich, timestamped textual commentaries that capture both factual and emotional
aspects of the game. Aiming to bridge the gap for fans unable to watch soccer
games live by providing engaging summaries that mimic the excitement of real-
time commentary, this task demonstrated the growing importance of textual
modality in creating immersive fan experiences.

Lashin et al. [19] highlighted the transformative potential of integrating ASR
alongside video and audio data in dense video captioning. Their multimodal
approach significantly improved event description accuracy, emphasizing the
vital contextual cues provided by ASR that often elude video-only analyses. By
combining visual information with speech data, their method demonstrated
substantial performance enhancements, showcasing the synergistic benefits of
leveraging multimodal inputs for comprehensive video understanding. Movie
Audio Descriptions (MAD) dataset offers a novel benchmark for video-language
grounding tasks, by aligning audio descriptions of mainstream movies with
video content [30]. The multimodal approach has since been embraced [28],
enriching the scope of video captioning and commentary generation [26, 27],
and illustrating the transformative impact of ASR technology in video content
analysis and processing.

###  2.4 Advancements in ASR Technologies

The development of LLM-based ASR technologies [36] further accelerated the use
of speech recognition in sports analytics. The GOAL dataset [26], derived from
SoccerNet videos, comprises 80 full-game videos transcribed into raw text
using the Azure ASR toolkit, then meticulously curated by English-speaking,
soccer-knowledgeable annotators. 20 games were selected after quality
filtering, and annotation tasks included proofreading, aligning text with
video timelines, and annotating knowledge entities, ensuring comprehensive
linguistic and visual analysis. Ikeda et al. [17] leveraged ASR alongside
Whisper [36] and a speech recognition model trained on the extensive Japanese
audio corpus ReazonSpeech [35] to transcribe commentary from 134 Japanese
Major League Baseball (MLB) highlight videos. To address noise challenges,
Whisper transcripts underwent manual labeling, while ReazonSpeech transcripts
were labeled using keyword techniques, thereby improving dataset quality and
reducing time overheads. These approaches underscore a growing trend in
leveraging ASR technology to create rich datasets for detailed examination of
both linguistic and visual elements in sports commentary.

###  2.5 Broader Applications and Future Directions

The integration of ASR technology into soccer video analysis has followed a
trajectory from enhancing basic action recognition to enabling sophisticated,
multimodal interpretations of complex events. The evolution of this technology
has not only improved the analytical capabilities of researchers but has also
significantly enhanced the viewing experience for fans worldwide. Beyond
soccer, the integration of ASR has also influenced broader video analysis
tasks. Hessel et al. [16] explored combining ASR and visual features to
generate captions for instructional videos, highlighting how ASR can aid in
distinguishing between similar actions through linguistic cues. Sattar et al.
[27] focused on identifying events in cricket games by combining commentary
text obtained from ASR with visual data and cues such as replays, bowler and
umpire positions.

These applications suggest potential future directions where ASR could be
integrated into various types of content beyond sports, enriching content
accessibility and understanding across domains. As ASR technology continues to
advance, its applications in video analysis are expected to expand, further
revolutionizing the field and providing deeper insights into the intricate
dynamics of media content. However, it is important to acknowledge that
despite the advancements, ASR technology is not flawless. Automated capture
introduces uncertainty and errors in the ground truth data. Nonetheless, its
cost-effectiveness and scalability make ASR a promising candidate for
widespread use.

##  3 Dataset Curation

Our comprehensive approach to curating a multilingual soccer commentary
dataset, which includes the automated transcription and translation of
commentaries, is illustrated in Figure 1. Below, we detail the curation of the
SoccerNet-Echoes using the illustrated methodology.

###  3.1 Sources and Collection

We used all 1100 game half videos from the SoccerNet [12] dataset. These
correspond to 550 games from 6 different leagues and 4 different seasons, as
depicted in Figure 2. The games were narrated live by commentators speaking
one of 10 different languages. The distribution of the original language of
the broadcast game audio is provided in Figure 3. We automatically extracted
commentaries in textual format following the procedure described below.

EPLLigue1SerieABundesligaLaLigaUCL$0$$20$$40$$60$$80$Number of
Games2014-20152015-20162016-20172019-2020 Figure 2: Number of games per league
and season in the SoccerNet dataset (550 in total).
EnglishSpanishRussianGermanFrenchTurkishItalianPolishBosnianHungarianNot
Available$0$$100$$200$$300$$297$$264$$218$$135$$102$$4$$4$$2$$2$$2$$70$Count
Figure 3: Language distribution of the original broadcast audio for each game
half, identified using Google Translate and the Whisper language detection
model (1100 in total).

###  3.2 Transcriptions

We used OpenAI’s Whisper large-v1111https://huggingface.co/openai/whisper-
large, large-v2222https://huggingface.co/openai/whisper-large-v2, and
large-v3333https://huggingface.co/openai/whisper-large-v3 ASR models to
transcribe the commentator speech from the game audio into text, with the
default parameters provided in the original GitHub implementation (commit:
ba3f3cd) [25]. Initially, the ASR models were used to process all games
regardless of the presence of audible human commentary. By default, Whisper
models recognize the audio language from the first $30$ seconds of the input,
and assume that the rest of the audio is in the same language. It should be
noted that the model’s capability to accurately identify entity names
(players, teams, stadiums, etc.) is limited. We performed explicit language
detection for all games using Whisper-v2, the resulting language distribution
is depicted in Figure 3. Figure 4 presents an example transcription using
Whisper from original game audio in English.

(a)

(b)

(c)

(d)

(e)

Automatic English Transcription Frame a to e: "Here is Coman. I wonder if the
referee’s going to book him for that because that looked like a blatant dive
really."

Figure 4: Example transcription using Whisper (original audio in English). Key
frames are shown to represent the corresponding video segment.

Manual validation of videos without commentary. As shown in Figure 3, 70 game
half videos in the SoccerNet dataset lack audio commentary and therefore had
empty ASR outputs. We manually inspected these assets to ensure that they do
not in fact contain commentary. Overall, 56 videos were confirmed not to have
game audio at all, and 14 videos were confirmed to have game audio but without
commentary (only stadium). The detailed list of videos in the SoccerNet
dataset without audio commentary is released along with the SoccerNet-Echoes
dataset as a spreadsheet.

###  3.3 Translations

We used the language detection output from Whisper to filter non-English
commentaries, and translate the non-English transcriptions into English using
Google Translate [13]. This choice was made due to Google Translate’s
accessibility, ease, and comprehensive language support. Google Translate
generally provides reliable translations, particularly for common languages
and everyday communication. However, its accuracy can vary depending on
factors such as text complexity and linguistic nuances. Figures 5 and 6
present example transcriptions using Whisper and translations to English using
Google Translate (from original game audio in German and Russian,
respectively).

(a)

(b)

(c)

(d)

(e)

Automatic German Transcription and English Translation Frame a to e
(Transcription): "Spiel für RB Leipzig, die Mannschaft spielt in weiß und rot
Borussia Dortmund im ersten" Frame a to e (Translation): "Game for RB Leipzig,
the team plays in white and red Borussia Dortmund in the first"

Figure 5: Example transcription using Whisper and translation to English using
Google Translate (original audio in German). Key frames are shown to represent
the corresponding video segment.

(a)

(b)

(c)

(d)

(e)

Automatic Russian Transcription and English Translation Frame a to e
(Transcription): "Ну что, похоже будет прямой удар. Мурель попадает в
Лоподулу, который, по его мнению, играл рукой. Но Лоподулу сразу подхватывает
мяч. Все-таки, да, подумав, подумав, Масимилиану Арате назначает штрафной. Он
послушал выгон помощника. Показывает Лоподулу желтую карточку." Frame a to e
(Translation): "Well, it looks like it’s going to be a direct hit. Murel ends
up in Lopodula, who, in his opinion, played with his hand. But Lopodulu
immediately picks up the ball. Still, yes, after thinking, after thinking, he
appoints Masimiliana Arata penalty. He listened to the assistant’s drive.
Shows Lopodul a yellow card."

Figure 6: Example transcription using Whisper and translation to English using
Google Translate (original audio in Russian). Key frames are shown to
represent the corresponding video segment.

###  3.4 Final Public Dataset

The SoccerNet-Echoes dataset and relevant code are publicly released on GitHub
under https://github.com/SoccerNet/sn-echoes. The dataset is organized as
multiple folders under the _Dataset_ directory, grouped by league, season, and
game. Figure 9 presents an overview of the directory structure. Each ASR JSON
file contains the transcribed (and translated where applicable) commentary
segments for each game half, organized under the "segments" key, with each
segment identified by a unique index within the respective half. Every segment
records the start and end time in seconds, which delineate the temporal
boundaries of the commentary, along with the transcribed speech, which
provides a textual representation of the commentary content. Figure 10
presents the structure of the ASR JSON files.

##  4 Evaluation

For evaluating the transcription and translations in the SoccerNet-Echoes
dataset, we conducted an analysis that uses Word Error Rate (WER), Character
Error Rate (CER), BLEU score, and word count as metrics. These metrics are
crucial for discerning linguistic richness and accuracy.

###  4.1 Comparison with Verified Transcriptions

To evaluate the transcription performance, we conducted a comparative analysis
using manually verified transcriptions from the GOAL dataset [26]. This
dataset comprises human-verified transcriptions from 40 game half videos
across 20 games with original commentary in English from the SoccerNet
dataset. We assessed the transcription performance of Whisper large-v1/v2/v3
on these 40 videos against ground truth transcriptions from the GOAL dataset
using WER, CER, and BLEU score as metrics. The results are presented in Table
1.

Model | WER | CER | BLEU
---|---|---|---
Whisper large-v1 | 0.443 | 0.261 | 54.50
Whisper large-v2 | 0.458 | 0.269 | 52.59
Whisper large-v3 | 0.551 | 0.341 | 47.97
Table 1: Transcription performance for different Whisper models [25] against
ground truth information from the GOAL dataset [26], averaged over the 20
games for which ground truth transcriptions are available.

###  4.2 Model Selection

We implemented a "Unique Word Count" heuristic as a criterion to identify the
most effective ASR model among the different Whisper versions. This approach
facilitated the selection of the best-performing model for each game half
based on the diversity of vocabulary in the transcriptions, and proved
particularly valuable as a heuristic for assessing model performance in
scenarios prone to repeated content, often manifesting as hallucinations
(erroneously repeated phrases in the transcription, which are not actually
present in the audio). This metric effectively captures the ability of ASR
models to generate diverse linguistic outputs, which is crucial for minimizing
the impact of such repetitions.

For each game half video, we selected the model with the highest unique word
count as the "best" model for further analysis and application. This selection
criterion is anchored on the premise that a higher diversity in word usage is
indicative of reduced hallucinatory repetitions and, by extension, a more
robust transcription performance in diverse acoustic environments. Table 2
presents the average unique word ratio (unique word count divided by the the
number of total words) for different Whisper models, averaged over the videos
for which the model was selected to be the "best". A pie chart depicting the
distribution of selected Whisper models is given in Figure 7.

Model | Number of Videos | Average Unique Word Ratio
---|---|---
Whisper large-v1 | 316 | 0.311
Whisper large-v2 | 231 | 0.316
Whisper large-v3 | 483 | 0.370
Mixed Selection | 1030 | 0.340
Table 2: Average unique word ratio (number of unique words / number of total
words) for different Whisper models. "Mixed Selection" represents the overall
weighted average across a total of 1030 game half videos with valid
commentary, using the "best" model for each video. Figure 7: Distribution of
the selected Whisper models, for a total of 1030 game half videos with valid
commentary.

###  4.3 Vocabulary Analysis

We conducted an in-depth examination of the linguistic patterns in the soccer
game commentaries to identify prevalent verb-noun pairs, which are key to
understanding the action dynamics that are described. Utilizing Natural
Language Processing (NLP) techniques, we parsed the transcriptions to extract
named entities and specific grammatical structures, focusing on the
interactions between players and teams. We then applied dependency parsing
models to determine the dominant actions by identifying root verbs and their
direct objects, providing clarity on the primary activities depicted.

We conducted a sunburst visualization to represent the frequency and
distribution of verb-noun combinations, displaying verbs as primary segments
with corresponding nouns as sub-segments. This visualization, exemplified for
one game half in Figure 8, not only highlights predominant actions but also
showcases vocabulary diversity. Such analyses offer insights into sports
broadcasting narrative styles and can guide improvements in automated
commentary systems.

(a) Ground truth transcription from GOAL dataset

(b) Whisper large-v1

(c) Whisper large-v3

Figure 8: Sunburst plots representing the frequency of verb-noun combinations
in the audio commentary transcription for one game half.

##  5 Discussion

###  5.1 Dataset Applications

The augmentation of the SoccerNet dataset to include ASR data significantly
enriches the multimodal nature of the dataset, enabling a broad range of
applications which can leverage the combined strengths of video, audio, and
now textual data derived from spoken commentary. These applications are
fundamentally enabled by enhanced content understanding, i.e., the possibility
to train automated systems which can more effectively understand, interpret,
and generate soccer-related multimedia content, including an increased
contextual awareness for actions, in-game events, highlights/statistics, and
entities (e.g., teams, players, locations). Below, we discuss several
potential applications of the SoccerNet-Echoes dataset.

Multi-modal event detection: The integration of ASR data can enable more
sophisticated and accurate event detection mechanisms. Combining audio cues
(such as crowd noise and commentator excitement) and textual triggers from the
transcription (such as specific phrases indicating goals or fouls) with visual
data can improve the precision and recall of event detection. Such integration
is pivotal for real-time analytics, automated highlight generation, and
conducting more detailed statistical analysis. The transcription example
depicted in Figure 5 showcases the advantages of ASR in this context. In
scenarios where the sole analysis of video frames (e.g., based on the zoomed-
out shots of the soccer pitch as shown in Figures 5(a)-5(e)) might fail to
yield detailed information, ASR can provide contextual information, and allow
for the extraction of additional details (such as the identification of the
teams, and whether they are wearing home or away jerseys), which might not be
discernible from the visual data alone.

Game summarization: Automated systems can generate insightful and context-rich
summaries of soccer games using the textual modality. This involves not just
identifying key events, but also understanding their significance within the
game’s narrative, as well as the entities involved. For example, the frequency
with which a player’s name is mentioned during a crucial moment in the game
can indicate the level of their influence, which can be used for assigning an
appropriate amount of the overall word budget for the player in the
description of the action.

Automated soccer commentator: Commentary datasets such as SoccerNet-Echoes up
exciting possibilities for the development of automated real-time soccer game
commentator systems. With increasingly more datasets available for training,
models can be trained to create more realistic and contextually relevant, but
also dynamic and captivating commentary, potentially offering elevated levels
of immersion and engagement for the fans.

Tactical analysis: In certain cases, the ASR data can also provide a context
for tactical analysis, by enabling an understanding of the discussions made by
commentators based on the tactical adjustments made by the coaches during
soccer games. In this sense, spoken commentary provides a layer of strategic
insight that is typically not captured by video and audio alone. These
insights can then also be integrated with various audio cues (such as the
audio intensity levels from the stadium) to pinpoint and map tactical
highlights to important fan moments within the game.

These applications demonstrate the significant potential of the SoccerNet-
Echoes dataset to contribute to sports research and broadcasting. Using the
power of multimodal data, researchers and developers can create more
sophisticated tools that address a wide range of needs within the sports
community.

###  5.2 Limitations and Future Work

Our use of deep learning models for ASR and batch translation for the curation
of the SoccerNet-Echoes dataset have also introduced a number of limitations.
We aim to address several of the following challenges in future work.

Transcription accuracy: The inference capability and accuracy of the ASR
models are inherently constrained by their design, occasionally leading to
errors in transcription.

Hallucinations: Whisper models are known to be prone to hallucinations
(unwarranted repetition of phrases and words). During the curation of the
SoccerNet-Echoes dataset, we have observed hallucinations across all Whisper
ASR models, especially under conditions where the audio inputs were devoid of
human speech, were excessively noisy, or contained musical elements. Such
conditions significantly challenge transcription accuracy of the models,
leading to the anomalous generation of repeated phrases. These repetitions not
only degrade the quality of the transcription, but also affect the usability
of the transcribed text in downstream applications, such as subtitle
generation or voice-driven gameplay interaction. By analyzing the occurrences
of repeated words under various audio conditions, we aimed to understand the
strengths and limitations of different models.

Audio quality: As mentioned above, we have seen that the quality of the audio
to be transcribed significantly impacts the reliability of the transcriptions.
Implementing advanced audio pre-processing techniques to filter out background
noise and music could enhance the clarity of the input signal for ASR systems.
Combining Whisper with other technologies such as Voice Activity Detection
(VAD) could also help reduce ASR errors and hallucinations.

Human verification: The absence of human-verified annotations to serve as
ground truth poses a substantial challenge for all ASR datasets. Generating
such annotations is prohibitively expensive, thus limiting the ability of
researchers to comprehensively verify and refine ASR datasets. However, this
limitation is also critical to address, in order to advance the reliability
and applicability of our SoccerNet-Echoes dataset. We therefore plan to invest
in the curation of human-verified annotations, for a subset of the assets in
SoccerNet-Echoes, as future work. Despite being expensive, we believe that
such a dataset could provide a valuable benchmark for assessing and improving
ASR accuracy. Efforts could also involve collaborations with various academic
institutions and community-sourcing the annotations.

Batch translations: Machine translation systems, such as Google Translate, are
not infallible and their limitations must be acknowledged. When translations
are performed on entire texts at once, they can produce different outputs
compared to translations of specific, smaller segments. For instance, as
depicted in Figure 6, the Russian term "штрафной" was erroneously translated
as "penalty kick" when the entire game half was translated, contrasting with
the accurate translation "free kick" when only the segment was translated.
This discrepancy underscores the contextual challenges faced by machine
translation.

##  6 Conclusion

The augmentation of the SoccerNet dataset using ASR technology marks a pivotal
advance in the domain of sports analytics, providing a richer and more
integrated approach to understanding soccer games. The inclusion of multimodal
data not only aids in the accurate detection and description of in-game events
but also enriches the dataset’s applicability across various analytical tasks
such as sentiment analysis and tactical assessments. Despite the limitations
associated with ASR, such as potential inaccuracies and hallucinations under
challenging audio conditions, the benefits presented by the SoccerNet-Echoes
dataset are substantial. Future work will focus on refining ASR accuracy
through the curation of a human-verified annotation subset and the
implementation of advanced audio pre-processing techniques. By continually
improving the dataset’s quality and the methodologies used to analyze it,
SoccerNet-Echoes is poised to significantly influence the development of
automated systems for sports broadcasting and analytics, enhancing the
consumption and comprehension of sports events globally. This work not only
underscores the transformative impact of integrating ASR into sports video
analysis but also sets a precedent for future research in the field, promising
deeper insights and more effective tools for researchers, sports analytics
professionals, and enthusiasts.

## Acknowledgement

This work was partly funded by the Research Council of Norway, project number
346671 (AI-Storyteller), and has benefited from the Experimental
Infrastructure for Exploration of Exascale Computing (eX3), which is
financially supported by the Research Council of Norway under contract 270053.

## References

  * Barnard et al. [2003] M. Barnard, J.-M. Odobez, and S. Bengio.  Multi-modal audio-visual event recognition for football analysis.  In _IEEE XIII Workshop on Neural Networks for Signal Processing (IEEE Cat. No.03TH8718)_ , pages 17–19. IEEE, 2003.  ISBN 978-0-7803-8177.  doi: 10.1109/NNSP.2003.1318046.
  * Cabado et al. [2024] Bruno Cabado, Anthony Cioppa, Silvio Giancola, Andrés Villa, Bertha Guijarro-Berdinas, Emilio J Padròn, Bernard Ghanem, and Marc Van Droogenbroeck.  Beyond the premier: Assessing action spotting transfer capability across diverse domains.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), CVsports_ , 2024.
  * Chang et al. [1996] Yuh-Lin Chang, Wenjun Zeng, I. Kamel, and R. Alonso.  Integrated image and speech analysis for content-based video indexing.  In _Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems_ , pages 17–23. IEEE, 1996.  ISBN 978-0-8186-7438.  doi: 10.1109/MMCS.1996.534992.
  * Cioppa et al. [2022a] Anthony Cioppa, Adrien Deliège, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck.  Scaling up SoccerNet with multi-view spatial localization and re-identification.  _Sci. Data_ , 9(1):1–9, Jun. 2022a.  doi: 10.1038/s41597-022-01469-1.  URL https://doi.org/10.1038/s41597-022-01469-1.
  * Cioppa et al. [2022b] Anthony Cioppa, Silvio Giancola, Adrien Deliege, Le Kang, Xin Zhou, Zhiyu Cheng, Bernard Ghanem, and Marc Van Droogenbroeck.  SoccerNet-tracking: Multiple object tracking dataset and benchmark in soccer videos.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , pages 3490–3501, New Orleans, LA, USA, Jun. 2022b. Inst. Electr. Electron. Eng. (IEEE).  doi: 10.1109/cvprw56347.2022.00393.  URL https://doi.org/10.1109/CVPRW56347.2022.00393.
  * Cioppa et al. [2023] Anthony Cioppa, Silvio Giancola, Vladimir Somers, Floriane Magera, Xin Zhou, Hassan Mkhallati, et al.  SoccerNet 2023 Challenges Results.  _arXiv_ , Sept. 2023.  doi: 10.48550/arXiv.2309.06006.
  * Deliège et al. [2021] Adrien Deliège, Anthony Cioppa, Silvio Giancola, Meisam J. Seikavandi, Jacob V. Dueholm, Kamal Nasrollahi, et al.  SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 19–25. IEEE, 2021.  doi: 10.1109/CVPRW53098.2021.00508.
  * Gao et al. [2020] Xin Gao, Xusheng Liu, Taotao Yang, Guilin Deng, Hao Peng, Qiaosong Zhang, et al.  Automatic Key Moment Extraction and Highlights Generation Based on Comprehensive Soccer Video Understanding.  In _2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_, pages 06–10. IEEE, 2020.  doi: 10.1109/ICMEW46912.2020.9106051.
  * Gautam [2023] Sushant Gautam.  Bridging Multimedia Modalities: Enhanced Multimodal AI Understanding and Intelligent Agents.  In _ICMI ’23: Proceedings of the 25th International Conference on Multimodal Interaction_ , pages 695–699. Association for Computing Machinery, New York, NY, USA, Oct. 2023.  ISBN 979-840070055-2.  doi: 10.1145/3577190.3614225.
  * Gautam et al. [2022a] Sushant Gautam, Cise Midoglu, Saeed Shafiee Sabet, Dinesh Baniya Kshatri, and Pl Halvorsen.  Soccer Game Summarization using Audio Commentary, Metadata, and Captions.  In _NarSUM ’22: Proceedings of the 1st Workshop on User-centric Narrative Summarization of Long Videos_ , pages 13–22. Association for Computing Machinery, New York, NY, USA, Oct. 2022a.  ISBN 978-1-45039493-2.  doi: 10.1145/3552463.3557019.
  * Gautam et al. [2022b] Sushant Gautam, Cise Midoglu, Saeed Shafiee Sabet, Dinesh Baniya Kshatri, and Pl Halvorsen.  Assisting soccer game summarization via audio intensity analysis of game highlights.  In _Proc. 12th IOE Graduate Conference_ , pages 25–32. Inst. of Eng. Tribhuvan University, Nepal, Oct. 2022b.  URL http://conference.ioe.edu.np/publications/ioegc12/IOEGC-12-004-12009.pdf.
  * Giancola et al. [2018] Silvio Giancola, Mohieddine Amine, Tarek Dghaily, and Bernard Ghanem.  SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 18–22. IEEE, 2018.  doi: 10.1109/CVPRW.2018.00223.
  * Google [2024] Google.  Google Translate, Apr. 2024.  [Online; https://translate.google.com].
  * Held et al. [2023] Jan Held, Anthony Cioppa, Silvio Giancola, Abdullah Hamdi, Bernard Ghanem, and Marc Van Droogenbroeck.  VARS: Video assistant referee system for automated soccer decision making from multiple views.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW)_ , pages 5086–5097, Vancouver, Can., Jun. 2023. Inst. Electr. Electron. Eng. (IEEE).  doi: 10.1109/cvprw59228.2023.00537.  URL https://doi.org/10.1109/CVPRW59228.2023.00537.
  * Held et al. [2024] Jan Held, Hani Itani, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck.  X-vars: Introducing explainability in football refereeing with multi-modal large language models.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , Seattle, WA, USA, Jun. 2024.
  * Hessel et al. [2019] Jack Hessel, Bo Pang, Zhenhai Zhu, and Radu Soricut.  A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions.  _ACL Anthology_ , pages 419–429, Nov. 2019.  doi: 10.18653/v1/K19-1039.
  * Ikeda et al. [2024] Riku Ikeda, Kazuma Sakamoto, and Yoshihiro Ueda.  Breaking News System of At-Bat Results From Sports Commentary via Speech Recognition.  _IEEE Access_ , 12:27199–27209, Feb. 2024.  ISSN 2169-3536.  doi: 10.1109/ACCESS.2024.3365948.
  * Krishna et al. [2017] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-Captioning Events in Videos.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 22–29. IEEE, 2017.  doi: 10.1109/ICCV.2017.83.
  * Lashin and Rahtu [2020] Vladimir Lashin and Esa Rahtu.  _Multi-modal Dense Video Captioning_.  IEEE Computer Society, Jun. 2020.  ISBN 978-1-7281-9360-1.  doi: 10.1109/CVPRW50498.2020.00487.
  * Li [2022] Jinyu Li.  Recent Advances in End-to-End Automatic Speech Recognition.  _SIP_ , 11(1), Apr. 2022.  doi: 10.1561/116.00000050.
  * Lu and An [2020] Yao Lu and Shuyang An.  Research on sports video detection technology motion 3D reconstruction based on hidden Markov model.  _Cluster Comput._ , 23(3):1899–1909, Sept. 2020.  ISSN 1573-7543.  doi: 10.1007/s10586-020-03097-z.
  * Malik et al. [2021] Mishaim Malik, Muhammad Kamran Malik, Khawar Mehmood, and Imran Makhdoom.  Automatic speech recognition: a survey.  _Multimed. Tools Appl._ , 80(6):9411–9457, Mar. 2021.  ISSN 1573-7721.  doi: 10.1007/s11042-020-10073-7.
  * Mkhallati et al. [2023a] Hassan Mkhallati, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck.  SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 17–24. IEEE, 2023a.  doi: 10.1109/CVPRW59228.2023.00536.
  * Mkhallati et al. [2023b] Hassan Mkhallati, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck.  SoccerNet-caption: Dense video captioning for soccer broadcasts commentaries.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW)_ , pages 5074–5085, Vancouver, Can., Jun. 2023b. Inst. Electr. Electron. Eng. (IEEE).  doi: 10.1109/cvprw59228.2023.00536.  URL https://doi.org/10.1109/CVPRW59228.2023.00536.
  * OpenAI [2024] OpenAI.  Whisper.  GitHub, Apr. 2024.  [Online; https://github.com/openai/whisper].
  * Qi et al. [2023] Ji Qi, Jifan Yu, Teng Tu, Kunyu Gao, Yifan Xu, Xinyu Guan, et al.  GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generation.  In _CIKM ’23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , pages 5391–5395. Association for Computing Machinery, New York, NY, USA, Oct. 2023.  ISBN 979-840070124-5.  doi: 10.1145/3583780.3615120.
  * Sattar et al. [2023] Husnain Sattar, Muhammad Shamil Umar, Eeman Ijaz, and Muhammad Umair Arshad.  Multi-Modal Architecture for Cricket Highlights Generation: Using Computer Vision and Large Language Model.  In _2023 17th International Conference on Open Source Systems and Technologies (ICOSST)_ , pages 20–21. IEEE, 2023.  doi: 10.1109/ICOSST60641.2023.10414235.
  * Seo et al. [2022] Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid.  _End-to-end Generative Pretraining for Multimodal Video Captioning_.  IEEE Computer Society, Jun. 2022.  ISBN 978-1-6654-6946-3.  doi: 10.1109/CVPR52688.2022.01743.
  * Shih [2017] Huang-Chia Shih.  A Survey of Content-Aware Video Analysis for Sports.  _IEEE Trans. Circuits Syst. Video Technol._ , 28(5):1212–1231, Jan. 2017.  doi: 10.1109/TCSVT.2017.2655624.
  * Soldan et al. [2022] Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem.  Mad: A scalable dataset for language grounding in videos from movie audio descriptions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5026–5035, June 2022.
  * Somers et al. [2024] Vladimir Somers, Victor Joos, Silvio Giancola, Anthony Cioppa, Seyed Abolfazl Ghasemzadeh, Floriane Magera, Baptiste Standaert, Amir Mohammad Mansourian, Xin Zhou, Shohreh Kasaei, Bernard Ghanem, Alexandre Alahi, Marc Van Droogenbroeck, and Christophe De Vleeschouwer.  SoccerNet game state reconstruction: End-to-end athlete tracking and identification on a minimap.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , Seattle, WA, USA, Jun. 2024.
  * Soomro et al. [2012] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild.  _arXiv_ , Dec. 2012.  doi: 10.48550/arXiv.1212.0402.
  * Szeliski [2022] Richard Szeliski.  _Computer Vision_.  Springer International Publishing, Cham, Switzerland, 2022.  ISBN 978-3-030-34372-9.  URL https://link.springer.com/book/10.1007/978-3-030-34372-9.
  * Vanderplaetse and Dupont [2020] Bastien Vanderplaetse and Stéphane Dupont.  Improved Soccer Action Spotting using both Audio and Video Streams.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 14–19. IEEE, 2020.  doi: 10.1109/CVPRW50498.2020.00456.
  * Yin et al. [2023] Yue Yin, Daijiro Mori, and Seiji Fujimoto.  Reazonspeech: A free and massive corpus for japanese asr.  In _Proceedings of the 29th Annual Conference of the Language Processing Society (NLP 2023)_ , March 2023.
  * Zhang et al. [2018] Zixing Zhang, Jürgen Geiger, Jouni Pohjalainen, Amr El-Desoky Mousa, Wenyu Jin, and Björn Schuller.  Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments.  _ACM Trans. Intell. Syst. Technol._ , 9(5):1–28, Apr. 2018.  ISSN 2157-6904.  doi: 10.1145/3178115.

##  Appendix A APPENDIX

Figure 9 presents an overview of the SoccerNet-Echoes dataset directory under
https://github.com/SoccerNet/sn-echoes.

.1 Dataset. .2 whisper_v1. .3 england_epl. .4 2014-2015. .5 2016-03-02-23-00
Liverpool 3-0 Manchester City. .6 1_asr.json. .6 2_asr.json. .5 … . .4
2015-2016. .4 … . .3 europe_uefa-champions-league. .4 … . .3 france_ligue-1.
.4 … . .3 germany_bundesliga. .4 … . .3 italy_serie-a. .4 … . .3 spain_laliga.
.4 … . .2 whisper_v1_en. .3 … . .2 whisper_v2. .3 … . .2 whisper_v2_en. .3 … .
.2 whisper_v3. .3 … . .2 whisper_v3_en. .3 … .

Figure 9: SoccerNet-Echoes dataset directory structure.

Figure 10 presents the structure of the ASR JSON files in the dataset.

⬇

{

"segments": {

(int) <segment index>:[

(float) <start time in seconds>,

(float) <end time in seconds>,

(string) <commentary text>

],

....

}

}

Figure 10: Structure of the ASR JSON files in the SoccerNet-Echoes dataset.