# Efficient Reinforcement Learning in Resource Allocation Problems Through
Permutation Invariant Multi-task Learning

Desmond Cai,1 Shiau Hong Lim2, Laura Wynter2 1Desmond Cai is with AStar,
Singapore<EMAIL_ADDRESS>Hong Lim and Laura Wynter are with IBM
Research, Singapore<EMAIL_ADDRESS>
###### Abstract

One of the main challenges in real-world reinforcement learning is to learn
successfully from limited training samples. We show that in certain settings,
the available data can be dramatically increased through a form of multi-task
learning, by exploiting an invariance property in the tasks. We provide a
theoretical performance bound for the gain in sample efficiency under this
setting. This motivates a new approach to multi-task learning, which involves
the design of an appropriate neural network architecture and a prioritized
task-sampling strategy. We demonstrate empirically the effectiveness of the
proposed approach on two real-world sequential resource allocation tasks where
this invariance property occurs: financial portfolio optimization and meta
federated learning.

##  1 Introduction

Sample efficiency in reinforcement learning (RL) is an elusive goal. Recent
attempts at increasing the sample efficiency of RL implementations have
focused to a large extent on incorporating models into the training process:
[25, 6, 28, 3, 12, 26, 9, 5, 20]. The models encapsulate knowledge explicitly,
complementing the experiences that are gained by sampling from the RL
environment. Another means towards increasing the availability of samples for
a reinforcement learner is by tilting the training towards one that will
better transfer to related tasks: if the training process is sufficiently well
adapted to more than one task, then the training of a particular task should
be able to benefit from samples from the other related tasks. This idea was
explored a decade ago in [13] and has been gaining traction ever since, as
researchers try to increase the reach of deep reinforcement learning from its
comfortable footing in solving games outrageously well to solving other
important problems. [27] discusses a number of methods for increasing sample
efficiency in RL and includes experience transfer as one important avenue,
covering the transfer of samples, as we do here, transfer of representation or
skills, and jumpstarting models which are then ready to be quickly, i.e. with
few samples, updated to different tasks. [7] address the same idea, noting
that multi-task learning can improve the learning of each individual task,
motivated by robotics-type tasks with underlying commonality, such as
balancing a single vs. a double pendulum, or hopping vs. walking.

We are interested in exploiting the ability of multi-task learning to solve
the sample efficiency problem of RL. Our setting does not apply to all problem
classes nor does it seek to exploit the kind of physical similarities found in
robotics tasks that form the motivation of [13, 7]. Rather, we show that there
are a number of reinforcement learning tasks with a particular fundamental
property that makes them ideal candidates for multi-task learning with the
goal of increasing the availability of samples for their training. We refer to
this property as permutation invariance. It is present in very diverse tasks:
we illustrate it on a financial portfolio optimization problem, whereby trades
are executed sequentially over a given time horizon, and on the problem of
meta-learning in a federated supervised learning setting.

Permutation invariance in the financial portfolio problem exhibits itself as
follows: consider the task of allocating a portion of wealth to each of a
number of financial instruments using a trading policy. If the trading policy
is permutation invariant, one can change the order of the instruments without
changing the policy. This allows one to generate multiple portfolio
optimization tasks from a given set of financial instruments. A commonality
between applications that have this property is that they concern sequential
resource allocation: at each time step, the resource allocation scores the
quality of each available candidate entity (for example a financial instrument
in the above example), then based on those scores, apportions out the resource
(the total wealth to invest, in the above example) among the entities at that
time step, so that over the horizon of interest, the reward is maximized.

Sequential resource allocation problems include applications such as
sequential allocation of budget, sequential allocation of space, e.g. in IT
systems, hotels, delivery vehicles, sequential allocation of people to work
slots or appointments, etc. Many such applications possess permutation
invariance in that the ordering of the entities, i.e. where the resources are
allocated, can change without changing the resulting optimal allocation. We
show that under this form of permutation invariance, it is possible to derive
a bound on the performance of the policy. The bound is an extension of that of
[13], and while similar to, provides additional information beyond the bound
of [7]. We use the bound to motivate an algorithm that allows for
substantially improved results as compared with solving each task on its own.
The bound and the algorithm are first analyzed on a synthetic problem that
validates the bound in our theorem and confirms the multi-task gain that the
theory predicts. [8, 4] have cautioned against degrading of the performance on
each task when some tasks bias the updates to the detriment of others in
multi-task learning. They claim that some tasks have a greater density or
magnitude of in-task rewards and hence a disproportionate impact on the
learning process. In our setting, deleterious effects of some tasks on others
could also arise. The algorithm we propose handles this through a form of
prioritized sampling, where priorities are put on the tasks themselves, and
acts like a prioritized experience replay buffer, applied to a multi-task
learning problem. We show empirically that the priorities thus defined protect
the overall learning problem from the deleterious effects that unrelated or
unhelpful tasks could otherwise have on the policy.

The contributions of this work are as follows: (1) we identify the permutation
invariance property of the class of reinforcement learning problems involving
sequential resource allocation, (2) we define a method to increase sample
efficiency in these reinforcement learning problems by leveraging this
property of permutation invariance; (3) we provide a theoretical performance
bound for the class of problems; (4) we validate experimentally the utility of
permutation variance on sample efficiency as well as the validity of the bound
on a synthetic problem; and (5) we illustrate two real-world RL resource
allocation tasks for which this property holds and demonstrate the benefits of
the proposed method on sample efficiency and thus also on the overall
performance of the models.

##  2 Related work

A notable first stream of work on leveraging multi-task learning for enhancing
RL performance on single tasks can be found in [24, 13] which consider, as we
do, that there is an underlying MDP from which the multiple tasks can be
thought to derive. They use however a Bayesian approach and propose a
different algorithmic method than ours. Our results extend performance bounds
by [14] on single-task RL. As noted by [27], jumpstarting, or distilling
experiences and representations of relevant policies is another means to
increasing sample efficiency in solving a new but related problem. [18] uses
this idea in so-called progressive neural networks and [17] leverage multiple
experts to guide the derivation of a general policy. With a similar objective,
[21] define a policy centroid, that is, a shared distilled policy, that
captures the commonalities across the behaviors in the tasks. In all of these
distillation-type methods, the tasks considered are simple or complex games.

[21] note that their policy centroid method, distral, is likely to be affected
by task interference, in that differences across tasks may degrade the
performance of the resulting policy of any of the constituent tasks. This
topic was studied by [8, 4]. [8] proposed a solution to this by extending the
so-called PopArt normalization [22] to re-scale the updates of each task so
that the different characteristics of the task-specific reward do not skew the
learning process. [4] use a different approach that learns attention weights
of the sub-networks of each task and discards those that are not relevant or
helpful. [23, 7] are, like our work, concerned with sharing of experiences to
facilitate a more sample-efficient learning process. [23] suggest identifying
the shared portions of tasks to allow sharing of samples in those portions.
The work of [7] is in some ways quite similar to ours: the authors’ goal is
the same and they derive a bound as we do on the performance in this setting.
However, their setting is different in that their tasks have both shared and
task-specific components, and their bound becomes tighter only as the number
of tasks increases. In our setting, we do not require a task-specific
component, and we are able to show how the distance between the MDPs of each
task, in addition to the number of tasks, affects the strength of the bound.
Recently, permutation invariance has been exploited in deep multi-agent
reinforcement learning [15] where the invariance properties arise naturally in
a homogeneous multi-agent setting. Their work employs permutation invariance
in learning the critic whereas in our case the entire learned policy employs
permutation invariance.

##  3 Preliminaries

We begin by defining notation. For a measurable space with domain
$\mathcal{X}$, let $\mathcal{S}(\mathcal{X})$ denote the set of probability
measures over $\mathcal{X}$, and $\mathcal{B}(\mathcal{X};L)$ the space of
bounded measurable functions with domain $\mathcal{X}$ and bound $0<L<\infty$.
For a measure $\rho\in\mathcal{S}(\mathcal{X})$ and a measurable function
$f:\mathcal{X}\rightarrow\mathbb{R}$, the $l_{2}(\rho)$-norm of $f$ is
$\|f\|_{\rho}$, and for a set of $n$ points
$X_{1},\cdots,X_{n}\in\mathcal{X}$, the empirical norm, $\|f\|_{n}$ is

$\displaystyle\|f\|_{\rho}^{2}=\int
f(x)^{2}\rho(dx)\quad\text{and}\quad\|f\|_{n}^{2}=\frac{1}{n}\sum_{t=1}^{n}f(X_{t})^{2}.$

Let $\|f\|_{\infty}=\sup_{x\in\mathcal{X}}|f(x)|$ be the supremum norm of $f$.
Consider a set of MDPs indexed by $t$. Each MDP is denoted by a tuple
$\mathcal{M}_{t}=\langle\mathcal{X},\mathcal{A},R_{t},P_{t},\gamma\rangle$,
where $\mathcal{X}$, a bounded closed subset of the $s$-dimensional Euclidean
space, is a common state space; $\mathcal{A}$ is a common action space,
$R_{t}:\mathcal{X}\times\mathcal{A}\rightarrow\mathbb{R}$ is a task specific
reward function uniformly bounded by $R_{\text{max}}$, $P_{t}$ is a task
specific transition kernel such that $P_{t}(\cdot|x,a)$ is a distribution over
$\mathcal{X}$ for all $x\in\mathcal{X}$ and $a\in\mathcal{A}$, and
$\gamma\in(0,1)$ is a common discount factor. Deterministic policies are
denoted by $\pi:\mathcal{X}\rightarrow\mathcal{A}$. For a given policy $\pi$,
the MDP $\mathcal{M}_{t}$ is reduced to a Markov chain
$\mathcal{M}_{t}^{\pi}=\langle\mathcal{X},R_{t}^{\pi},P_{t}^{\pi},\gamma\rangle$
with reward function $R_{t}^{\pi}(x)=R_{t}(x,\pi(x))$, transition kernel
$P_{t}^{\pi}(\cdot|x)=P_{t}(\cdot|x,\pi(x))$, and stationary distribution
$\rho_{t}^{\pi}$. The value function $V_{t}^{\pi}$ for MDP $t$ is defined as
the unique fixed-point of the Bellman operator
$\mathcal{T}_{t}^{\pi}:\mathcal{B}(\mathcal{X};V_{\text{max}}=R_{\text{max}}/(1-\gamma))\rightarrow\mathcal{B}(\mathcal{X};V_{\text{max}})$,
given by

$(\mathcal{T}_{t}^{\pi}V)(x)=R_{t}^{\pi}(x)+\gamma\int_{\mathcal{X}}P_{t}^{\pi}(dy|x)V(y).$

Let $\pi_{t}^{*}$ denote the optimal policy for $\mathcal{M}_{t}$. The optimal
value function $V_{t}^{\pi_{t}^{*}}$ for $\mathcal{M}_{t}$ is defined as the
unique fixed-point of its optimal Bellman operator
$\mathcal{T}_{t}^{\pi_{t}^{*}}$ which is defined by

$(\mathcal{T}_{t}^{\pi_{t}^{*}}V)(x)=\max_{a\in\mathcal{A}}\left[R_{t}(x,a)+\gamma\int_{\mathcal{X}}P_{t}(dy|x,a)V(y)\right].$

To approximate the value function $V$, we use a linear approximation
architecture with parameters $\alpha\in\mathbb{R}^{d}$ and basis functions
$\varphi_{i}\in\mathcal{B}(\mathcal{X};L)$ for $i=1,\cdots,d$. Let
$\varphi(\cdot)=(\varphi_{1}(\cdot),\cdots,\varphi_{d}(\cdot))^{\mathsf{T}}\in\mathbb{R}^{d}$
be the feature vector and $\mathcal{F}$ the linear function space spanned by
basis functions $\varphi_{i}$. Thus,
$\mathcal{F}=\\{f_{\alpha}\;|\;\alpha\in\mathbb{R}^{d}\;\text{and}\;f_{\alpha}(\cdot)=\varphi(\cdot)^{\mathsf{T}}\alpha\\}$.

Consider a learning task to dynamically allocate a common resource across
entities $\mathcal{U}_{t}\subseteq\mathcal{U}$. Each $t$ corresponds to a
task, but for now take $t$ to be an arbitrary fixed index. At each time step
$n$, the decision maker observes states
$x_{n}=(x_{i,n})_{i\in\mathcal{U}_{t}}$ of the entities, where $x_{i,n}$ is
the state of entity $i$, and takes action
$a_{n}=(a_{i,n})_{i\in\mathcal{U}_{t}}$, where $a_{i,n}$ is the share of the
resource allocated to entity $i$. The total resource capacity is normalized to
$1$ for convenience. Therefore, allocations satisfy $0\leq a_{i,n}\leq 1$ and
$\sum_{i\in\mathcal{U}_{t}}a_{i,n}=1$. We consider policy
$\pi_{\theta}(x_{n})$ parameterized by $\theta$. Assume that we have access to
the reward function $R_{t}$ as well as a simulator that generates a trajectory
of length $N$ given any arbitrary policy $\pi_{\theta}$. The objective of the
learning task is to maximize

$\displaystyle J_{t}(\theta)=\mathbb{E}$
$\displaystyle[\left.\sum_{n=1}^{N}\gamma^{n-1}R_{t}(x_{n},a_{n})\;\right|a_{n+1}=\pi_{\theta}(x_{n}),\;$
$\displaystyle x_{n+1}\sim P_{t}(\cdot|x_{n},a_{n}),\;x_{1}\sim P_{t}(\cdot)]$

In many settings, $N$ is small and simulators are inaccurate; therefore,
trajectories generated by the simulator are poor representations of the actual
transition dynamics. This occurs in batch RL where trajectories are rollouts
from a dataset. In these cases, policies overfit and generalize poorly.

##  4 Theoretical Results

We introduce first a property that we term permutation-invariance for the
policy network that can be shown to help significantly reduce overfitting.

######  Definition 1 (Permutation Invariant Policy Network)

A policy network $\pi_{\theta}$ is permutation invariant if it satisfies
$\pi_{\theta}(\sigma(x))=\sigma(\pi_{\theta}(x))$ for any permutation
$\sigma$.

Permutation invariant policy networks have significant advantages over
completely integrated policy networks. While the latter are likely to fit
correlations between different entities, this is not possible with permutation
invariant policy networks as they are agnostic to identities of entities.
Therefore, permutation invariant policy networks are better able to leverage
experience across time and entities, leading to greater efficiency in data
usage. Moreover, observe that if the transition kernels can be factored into
independent and identical transition kernels across entities, then the optimal
policy is indeed permutation invariant.

Our main theoretical contributions start with an extension of results from
[14], where a finite-sample error bound was derived for the least squares
policy iteration (LSPI) algorithm on a single task. [14] provided a high-
probability bound on the performance difference between the final learned
policy and the optimal policy, of the form $c_{1}+c_{2}/\sqrt{N}$, where
$c_{1}$ and $c_{2}$ are constants that depend on the task and the chosen
feature space, and $N$ is the number of training examples. We extend their
result by showing that, as long as tasks are $\epsilon$-close to each other
(with respect to a similarity measure we define later), the error bound of
solving each task using our multi-task approach has the form
$c_{1}+c_{2}/\sqrt{NT}+c_{3}\epsilon$, where $T$ is the number of tasks and
$c_{3}$ is a task-dependent constant. Specifically, our theorem provides a
general result and performance guarantee with respect to using data from a
different but similar MDP. Definition 1 provides a basis for generating many
such MDPs. Finally, the benefit of doing so shall be provided by Corollary 2.
Thus, provided $\epsilon$ is small, a given task can benefit from a much
larger set of $NT$ training examples.

In addition to the assumptions of [14], we extend the definition of second-
order discounted-average concentrability, proposed in [2], and define the
notion of first-order discounted-average concentrability. The latter will be
used in our main result, Theorem 1.

###### Assumption 1

There exists a distribution $\mu\in\mathcal{S}(\mathcal{X})$ such that for any
policy $\pi$ that is greedy with respect to a function in the truncated space
$\tilde{\mathcal{F}}$, $\mu\leq C\rho^{\pi}_{t}$ for all $t$, where $C<\infty$
is a constant. Given the target distribution
$\sigma\in\mathcal{S}(\mathcal{X})$ and an arbitrary sequence of policies
$\\{\pi_{m}\\}_{m\geq 1}$, let

$\displaystyle c_{\sigma,\mu}=\sup_{\pi_{1},\ldots,\pi_{m}}\left\|\frac{d(\mu
P^{\pi_{1}}\ldots P^{\pi_{m}})}{d\sigma}\right\|.$

We assume that $C_{\sigma,\mu}^{\prime},C_{\sigma,\mu}^{\prime\prime}<\infty$,
and define first and second order discounted-average concentrability of
future-state distributions as follows:

$\displaystyle C_{\sigma,\mu}^{\prime}$ $\displaystyle=(1-\gamma)\sum_{m\geq
0}\gamma^{m}c_{\sigma,\mu}(m),$ $\displaystyle C_{\sigma,\mu}^{\prime\prime}$
$\displaystyle=(1-\gamma)^{2}\sum_{m\geq 1}m\gamma^{m-1}c_{\sigma,\mu}(m).$

######  Theorem 1 (Multi-Task Finite-Sample Error Bound)

Let $\mathcal{M}=\langle\mathcal{X},\mathcal{A},R,P,\gamma\rangle$ be an MDP
with reward function $R$ and transition kernel $P$. Assume $\mathcal{A}$
finite. Denote its Bellman operator by

$\displaystyle(\mathcal{T}^{\pi}V)(x)=R^{\pi}(x)+\gamma\int_{\mathcal{X}}P^{\pi}(dy|x)V(y).$

Given a policy $\pi$, define the Bellman difference operator between
$\mathcal{M}_{t}$ and $\mathcal{M}$ to be
$\mathcal{D}_{t}^{\pi}V=\mathcal{T}_{t}^{\pi}V-\mathcal{T}^{\pi}V$. Apply the
LSPI algorithm to $\mathcal{M}$, by generating, at each iteration $k$, a path
from $\mathcal{M}$ of size $N$, where $N$ satisfies Lemma 4 in [14]. Let
$V_{-1}\in\tilde{\mathcal{F}}$ be an arbitrary initial value function,
$V_{0},\cdots,V_{K-1}$ ($\tilde{V}_{0},\cdots,\tilde{V}_{K-1}$) be the
sequence of value functions (truncated value functions) generated by the LSPI
after $K$ iterations, and $\pi_{k}$ be the greedy policy w.r.t. the truncated
value function $\tilde{V}_{k-1}$. Suppose also that

$\displaystyle\|\mathcal{D}_{t}^{\pi}V^{\pi}\|_{\mu}\leq\epsilon\;\forall\;\pi,\quad\text{and}\quad\|\mathcal{D}_{t}^{\pi_{k}}\tilde{V}_{k-1}\|_{\mu}\leq\epsilon\;\forall\;k.$

Then, for constants $c_{1}$, $c_{2}$, $c_{3}$, $c_{4}$ that are dependent on
$\mathcal{M}$, with probability $1-\delta$ (with respect to the random
samples):

$\displaystyle\|V_{t}^{\pi_{t}^{*}}-V_{t}^{\pi_{K}}\|_{\sigma}$
$\displaystyle\leq
c_{1}\frac{1}{\sqrt{N}}+c_{2}\epsilon\sqrt{C_{\sigma,\mu}^{\prime}}+c_{3}\sqrt{C_{\sigma,\mu}^{\prime\prime}}+c_{4}.$

The proof is deferred to the Appendix. Theorem 1 formalizes the trade off
between drawing fewer samples from the exact MDP $\mathcal{M}_{t}$, versus
drawing more samples from a different MDP $\mathcal{M}$. Importantly, it shows
how to benefit from solving a different MDP, $\mathcal{M}$, when: (a)
additional samples can be obtained from $\mathcal{M}$, and (b) $\mathcal{M}$
is not too different from $\mathcal{M}_{t}$. In particular, the distance
measure is simply the distance between the Bellman operators of the MDPs,
which can be bounded if the difference in both the transition and reward
functions are bounded.

In recent work, a performance bound for multi-task learning was given in
Theorem 2 and 3 of [7]. However, the authors used a different setup containing
both shared and task-specific representations, and their focus was on showing
that the cost of learning the shared representation decreases with more tasks.
They did not show how the similarity or difference across tasks affects
performance. In contrast, our setup does not contain task-specific
representations, and our focus is on how differences across MDPs impact the
benefit of having more tasks (and consequently more samples). We show this in
Corollary 1 and Corollary 2.

###### Remark 1

While our theoretical results are based on LSTD and LSPI and assume finite
action space, our approach is applicable to a wide range of reinforcement
learning algorithms, including policy gradient methods and to MDPs with
continuous action spaces. Deriving similar results for a larger family of
models and algorithms remains an interesting, albeit challenging, future work.

Permutation invariant policy networks allow using data from the global set of
entities $\mathcal{U}$. Since the policy network is agnostic to the identities
of the entities, one can learn a single policy for all tasks, where each task
$t\in[T]$ is a resource allocation problem over a subset of entities
$\mathcal{U}_{t}$. For notational simplicity, assume that all tasks have the
same number of entities, and all trajectories are of equal length $N$. Our
approach can, however, be readily extended to tasks with different numbers of
entities and different trajectory lengths. Permutation invariance allows a
large set of MDPs to leverage the result of Theorem 1. In the next section we
shall provide an algorithm, motivated by the following corollaries, and a
prioritized sampling strategy for this setting that drives significantly
greater sample efficiency for the original task. The sampling strategy also
helps to stabilize the learning process, reducing the risk of deleterious
effects of the multi-task setting, as discussed by [21] and addressed in works
such as [8, 4].

###### Corollary 1

Let $[T]$ be a set of similar tasks such that their distance from the average
MDP, given by

$\displaystyle(\mathcal{T}^{\pi}V)(x)$
$\displaystyle=\frac{1}{T}\sum_{t=1}^{T}R_{t}^{\pi}(x)+\gamma\int_{\mathcal{X}}\frac{1}{T}\sum_{t=1}^{T}P_{t}^{\pi}(dy|x)V(y),$

is bounded by $\epsilon$ as defined in Theorem 1. Let $N$ be the number of
samples available in each task. Let $\pi_{K}$ be the policy obtained at the
$K$th iteration when applying LSPI to the average MDP. Then, the suboptimality
of the policy on each task is $O(1/\sqrt{NT})+O(\epsilon)+c$ for some constant
$c$ (where suboptimality is defined according to Theorem 1).

Recall that each task is formed by selecting a subset $\mathcal{U}_{t}$ of
entities from the global set $\mathcal{U}$. We thus have the following sample
gain that can be attributed to the permutation invariance of the policy
network.

######  Corollary 2 (Sample Efficiency from Permutation Invar.)

Let $M=|\mathcal{U}|$ and $m=|\mathcal{U}_{t}|$. Given fixed $M$ and $m$,
there are $T={M\choose m}\geq\left(\frac{M}{m}\right)^{m}$ different tasks.
Then, by Cor. 1, assuming all pairs of tasks are weakly correlated, the
potential gain in sample efficiency is exponential in $m$.

Disregarding correlation between samples from tasks with overlapping entities
Corollary 1 and Corollary 2 together suggest that the (up to) exponential
increase in the number of available tasks can significantly improve sample
efficiency as compared to learning each task separately.

##  5 Exploiting Permutation Invariance through Multi-task Reinforcement
Learning

Our approach to exploiting permutation invariance is via multi-task
reinforcement learning, where each “task” corresponds to a particular choice
of subset $\mathcal{U}_{t}\subset\mathcal{U}$. Furthermore, for each task, we
enforce permutation invariance among the entities $i$ by forcing the neural
network to apply the same sequence of operations to the state input $x_{i}$ of
each instrument through parameter sharing.

The proposed method, shown in Algorithm 1, learns a single policy by sampling
subsequences of trajectories from the different MDPs. At each step, we sample
a task $t$ according to a distribution defined by task selection policy $p$.
Then, a minibatch sample $\mathcal{B}_{t}$ is drawn from the replay buffer for
task $t$, and gradient descent is performed using the sampled transitions
$\mathcal{B}_{t}$ (alternatively, samples can be generated using policy
rollouts for the specific task). Separate replay buffers maintained for each
task are updated only when the corresponding task is being used.

In contrast with other active sampling approaches in multi-task learning, our
approach maintains an estimate of the difficulty of each task $t$ as a score,
$s_{t}$. After each training step, we update the score for only the sampled
task based on minibatch $\mathcal{B}_{t}$, avoiding evaluation over all the
tasks. The scoring functions depend on the sampled minibatch; to reduce
fluctuations in scores for each task, exponential smoothing is applied
$s_{t}\leftarrow\gamma s_{t}+(1-\gamma)\cdot\mathsf{scorer}(\mathcal{B}_{t})$.
We propose a stochastic prioritization method that interpolates between pure
greedy prioritization and uniform random sampling. Our approach is similar to
prioritized experience replay (PER) by [19], but while classical PER
prioritizes samples, we prioritize tasks. The probability of sampling task $t$
is $p_{t}=s_{t}^{\alpha}/\sum_{t^{\prime}}s_{t^{\prime}}^{\alpha}$, where the
exponent $\alpha$ determines the degree of prioritization, with $\alpha=0$
corresponding to the uniform case. We correct for bias with importance-
sampling (IS) weights $w_{t}=1/(Tp_{t})^{\beta}$, that compensate for non-
uniform probabilities if $\beta=1$. We normalize weights by $1/\max_{t}w_{t}$.
Tasks on which the reward variance is high can be interpreted as having more
challenging samples, hence reward variance can be used as a scoring function.

Algorithm 1 Prioritized Multi-Task Reinforcement Learning for Increasing
Sample Efficiency

Initialize policy network $\pi_{\theta}$

Initialize replay buffers $R_{1},\ldots,R_{T}$

Initialize time steps $n_{1}\leftarrow 1,\ldots,n_{T}\leftarrow 1$

loop

Select a task $t\sim p$ to train on

Sample a random minibatch $\mathcal{B}_{t}$ of transitions
$(x_{n},a_{n},r_{n},x_{n+1})$ from $R_{t}$

Update policy $\theta$ using $\mathcal{B}_{t}$ and chosen RL approach
(correcting for bias using IS weights $w$)

Update score $s_{t}\leftarrow\gamma
s_{t}+(1-\gamma)\cdot\mathsf{scorer}(\mathcal{B}_{t})$

Update ALL selection probabilities $p$ and IS weights $w$

for $n=n_{t},\ldots,\min\\{n_{t}+n_{e},N\\}$ do

For task $t$, select action $a_{n}$ according to current policy and
exploration noise

Execute action $a_{n}$, and observe reward $r_{n}$ and new state $x_{n+1}$

Store transition $(x_{n},a_{n},r_{n},x_{n+1})$ in $R_{t}$

end for

If $n<N$, update $n_{t}\leftarrow n+1$, otherwise, update $n_{t}\leftarrow 1$

end loop

##  6 Experiments

###  6.1 Synthetic data

With the aim of validating the theory presented in Section 4, we define a
synthetic example to explore the efficiency gain afforded by permutation
invariance. To do so, we control of the deviation $\epsilon$ between any two
tasks, thereby empirically validating the main theoretical results.

Consider a resource allocation problem where the observed state $x_{i}$ for
each entity $i\in\\{1\ldots m\\}$ is a single scalar $x_{i}\in[0,1]$. The
action space is the probability simplex, where each action $a=(a_{1}\ldots
a_{m})$ indicates the fraction of resource allocated to each entity. The
reward function is

$R(x,a):=\sum_{i}x_{i}a_{i}-\beta_{i}a_{i}\log a_{i}$

where $\beta_{i}$ is a weight parameter for each entity. Note that when
$\beta_{i}=\beta$ for all $i$, the reward function becomes
$R(x,a)=(\sum_{i}x_{i}a_{i})+\beta H(a)$ where $H$ is the Shannon entropy.
This implies that maximizing the reward involves a tradeoff between focusing
resources on high $x_{i}$ or distributing them uniformly across all $i$. Note
that the reward function is permutation invariant, but that when we allow a
varying $\beta_{i}$ over the entities, the function deviates from being
perfectly permutation invariant. We use the range
$\max_{i}\beta_{i}-\min_{i}\beta_{i}$ as a stand-in for $\epsilon$.

Figure 1: Performance for $\epsilon=0.8$ (left), $\epsilon=0$ (middle), and at
$N=2000$ with varying $\epsilon$ (right).

Let $m=10$. For each $\epsilon$, we run two experiments. The first examines
the performance of policies trained by $LSPI$ using $N$ real examples drawn
i.i.d from the state-action space, for $N=20\ldots 2000$. A small Gaussian
noise is added to each reward to make learning harder. The second experiment
uses only $20$ real examples, but augments the training set (up to $N$)
through random permutation of the real examples. The first two figures in Fig.
1 show the results for $\epsilon=0.8$ and $\epsilon=0,$ respectively.
Performance improves with $N$, as predicted by the $1/\sqrt{N}$ term in our
error bound. Note that in the experiment using only $20$ real examples, a
performance gain is achieved by using permuted examples; this corresponds
precisely to the multi-task gain predicted by the $1/\sqrt{NT}$ term. When
$\epsilon$ is large, there is a significant gap between the results of the two
experiments, as predicted by the $\epsilon$-term in the error bound. The last
plot in Fig. 1 shows this gap at $N=2000$ when $\epsilon$ varies from 0 to
$0.8$.

###  6.2 Real-world data

We consider two real-world resource allocation settings: financial portfolio
optimization and meta federated learning. Financial portfolio optimization is
discussed below while meta federated learning is in the Appendix. Given
historical prices for a universe of financial assets, $\mathcal{U}$, the goal
of task $t$ is to allocate investments across a subset of assets
$\mathcal{U}_{t}\subseteq\mathcal{U}$. The multiple tasks $t$ thus correspond
to multiple portfolios of instruments. Permutation invariance will be of use
in this setting since, from a given universe of instruments (e.g. the 500
instruments in the S&P 500), an exponential number of tasks can be generated,
each with its own portfolio. Consider now one such task.

At the beginning of time period $n$, the action $a_{i,n}$ represents the
fraction of wealth the decision maker allocates to asset $i$. The allocations
evolve over the time period due to changes in asset prices. Let $w_{i,n}$
denote the allocation of asset $i$ at the end of time period $n$. We model the
state of an asset using its current allocation and a window of its $H$ most
recent prices. In particular, let $v_{i,n}$ denote the close price of asset
$i$ over time period $n$, and let $y_{i,n}=v_{i,n}/v_{i,n-1}$ denote the ratio
of close prices between adjacent time periods 111Daily high and low prices are
also used in the state but omitted here for brevity.. Then, the allocation in
asset $i$ at the end of time period $n$ is given by

$w_{i,n}=\frac{a_{i,n}y_{i,n}}{\sum_{i\in\mathcal{U}_{t}}a_{i,n}y_{i,n}},$

and the state of asset $i$ at the beginning of time period $n$ is given by

$x_{i,n}=(w_{i,n-1},v_{i,n-H}/v_{i,n-1},\ldots,v_{i,n-2}/v_{i,n-1}).$

The change in portfolio value over period $n$ depends on the asset prices and
transaction costs incurred in rebalancing the portfolio from
$(w_{i,n-1})_{i\in\mathcal{U}_{t}}$ to $(a_{i,n})_{i\in\mathcal{U}_{t}}$. The
reward over period $n$ is defined as the log rate of return:

$R_{t}(x_{n},a_{n})=\ln\left[\beta\left((w_{i,n-1})_{i\in\mathcal{U}_{t}},(a_{i,n})_{i\in\mathcal{U}_{t}}\right)\sum_{i\in\mathcal{U}_{t}}a_{i,n}y_{i,n}\right]$

where $\beta$ can be evaluated using an iterative procedure (see [10]).
Defining the reward this way is appealing because maximizing average total
reward over consecutive periods is equivalent to maximizing the total rate of
return over the periods. To leverage this, we approximate
$\beta((w_{i,n-1})_{i\in\mathcal{U}_{t}},(a_{i,n})_{i\in\mathcal{U}_{t}})\approx
c\sum_{i\in\mathcal{U}_{t}}|w_{i,n-1}-a_{i,n}|$, where $c$ is a commission
rate to obtain a closed-form expression for $R_{t}(x_{n},a_{n})$ (see [10]).
We optimize using direct policy gradient on minibatches of consecutive samples

$\theta\leftarrow\theta+\eta\nabla_{\theta}\left[\frac{1}{B}\sum_{n=n_{b}}^{n_{b}+B-1}w_{t}R_{t}(x_{n},\pi_{\theta}(x_{n}))\right],$

where $n_{b}$ is the first time index in the minibatch, $B$ the size of a
minibatch, and $w_{t}$ the IS weight for task $t$. As in [10], we sample
$n_{b}$ from a geometric distribution that prioritises recent samples and
implement replay buffers for each task. A benchmark trading strategy is equal
constantly-rebalanced portfolio (CRP) that rebalances to maintain equal
weights. As we noted earlier, ideally one would prefer for the scoring
function to depend only on the minibatch $\mathcal{B}_{t}$. A deviation from
Equal CRP can be viewed as  learning to exploit price movements, and is thus
here we use this as the goal of the policy. Prioritised MTL thus prioritises
tasks which deviate from Equal CRP. Note that the policy deviates from CRP
only when profitable. Let

$\mathsf{scorer}(\mathcal{B}_{t})=\max_{n\in\\{n_{b},\ldots,n_{b}+B-1\\}}\left\|\pi_{\theta}(x_{n})-\frac{1}{|\mathcal{U}_{t}|}\right\|_{\infty},$

be the scoring of tasks in Prioritised MTL using mean absolute deviation of
the minibatch allocation from Equal CRP. Figure 3 (left) shows a scatter plot
of the maximum score seen every $50$ steps and the change in episode rewards
in a single-task learning experiment, and (right) of the minibatch score and
the maximum gradient norm for the minibatch. Higher scores imply higher
variance in the episode rewards and hence more challenging and useful samples.
The correlation between scores and gradient norms shows that our approach is
performing gradient-based prioritisation, (see [11, 16, 1]) but in a
computationally efficient manner. The details of the dataset and parameter
settings can be found in the Appendix. Figure 3 shows the performance of the
learned policies tested on $10$ tasks drawn from out-of-sample instruments.
The policy network with weights initialized close to zero behaves like an
Equal CRP policy. As noted, any profitable deviation from Equal CRP implies
learning useful trading strategies. The plots show that the MTL policies
perform well on instruments never seen during training, offering a remarkable
benefit for using RL in the design of trading policies.

Figure 2: Scatter plots of the maximum absolute deviation from Equal CRP vs.
the change in rewards every $50$ steps (left) and the max. norm of the
gradient for the minibatch (right).

Figure 3: Mean performance gain over Equal CRP of the learned policies when
tested on $10$ tasks using out-of-sample instruments. Error bars denote the
standard deviation over $10$ experiments.

Fig. 4 shows the performance of prioritised multi-task learning (MTL) versus
single-task learning (STL) (i.e. learning a policy for each task independently
on the instruments in the task). We also show results for MTL without
prioritised sampling, i.e., with $\alpha=0$. We consider $5$ tasks and $30$
tasks. The plots show that prioritised MTL performs significantly better than
STL in both convergence time and final achieved performance. The performance
with $30$ tasks is significantly better than the performance with $5$ tasks,
showing that our approach leverages the samples of the additional tasks.

Fig. 5 illustrates the typical behavior of a multi-task learning (MTL) and a
single-task learning (STL) policy on the test period for tasks where multi-
task policy performed significantly better. The single-task policy kept
constant equal allocations while the multi-task policy was able to learn more
complex allocations. In financial data, strongly trending prices do not occur
often and are inherently noisy. Multi-task learning with permutation
invariance helps with both challenges, allowing the algorithm to learn more
complex patterns in a given training period.

Figure 4: The two left plots show mean annualized return in the testing period
over $10$ experiments (different instruments) each with $5$ and $30$ tasks.
X-axes are scaled to make the curves comparable: each epoch has $1500$
($5$-tasks) and $9000$ steps ($30$-tasks) and an evaluation. Shaded regions
denote the interquartile range. The rightmost figure shows, for each fraction
of tasks, the gain over Single Task Learning (STL). A curve further to the
right shows higher gain over STL. From 30% on the y-axis, the P-MTL gain is
higher (more towards the right) than the MTL gain. As expected, when few tasks
are used, prioritizing tasks doesn’t help much (y-axis from 0 to 0.2).

Figure 5: Comparison of a multi-task policy vs. a single-task policy on the
testing period for a specific task. The leftmost plot shows the percentage
gain in portfolio value over time for both policies against that from the
baseline Equal CRP policy. The right two plots show the asset allocations.

##  7 Conclusions

We introduce an approach for increasing the sample efficiency of reinforcement
learning in a setting with widespread applicability within the class of
sequential resource allocation problems. This property is permutation
invariance: resources are allocated to entities according to a score, and the
order can change without modifying the optimal allocation. Under this
property, we show that a bound exists on the policy performance. This bound
motivates a highly effective algorithm for improving the policy through a
multi-task approach. Using prioritized task-sampling, the method not only
improves the reward of the final policy but also renders it more robust. We
illustrate the property and the method on two important problems: sequential
financial portfolio optimization and meta federated learning, where the latter
is provided in the Appendix.

## Appendix

#### Theorem 1

Let $\mathcal{M}=\langle\mathcal{X},\mathcal{A},R,P,\gamma\rangle$ be an MDP
with reward function $R$ and transition kernel $P$. Denote its Bellman
operator by

$\displaystyle(\mathcal{T}^{\pi}V)(x)=R^{\pi}(x)+\gamma\int_{\mathcal{X}}P^{\pi}(dy|x)V(y).$

Given a policy $\pi$, define the Bellman difference operator between
$\mathcal{M}_{t}$ and $\mathcal{M}$ to be
$\mathcal{D}_{t}^{\pi}V=\mathcal{T}_{t}^{\pi}V-\mathcal{T}^{\pi}V$. Apply the
LSPI algorithm to $\mathcal{M}$, by generating, at each iteration $k$, a path
from $\mathcal{M}$ of size $N$, where $n$ satisfies Lemma 4 in [2]. Let
$V_{-1}\in\tilde{\mathcal{F}}$ be an arbitrary initial value function,
$V_{0},\cdots,V_{K-1}$ ($\tilde{V}_{0},\cdots,\tilde{V}_{K-1}$) be the
sequence of value functions (truncated value functions) generated by the LSPI
after $K$ iterations, and $\pi_{k}$ be the greedy policy w.r.t. the truncated
value function $\tilde{V}_{k-1}$. Suppose also that

$\displaystyle\|\mathcal{D}_{t}^{\pi}V^{\pi}\|_{\mu}\leq\epsilon\;\forall\;\pi,\quad\text{and}\quad\|\mathcal{D}_{t}^{\pi_{k}}\tilde{V}_{k-1}\|_{\mu}\leq\epsilon\;\forall\;k.$

Then, with probability $1-\delta$ (with respect to the random samples), we
have

$\displaystyle\|V_{t}^{\pi_{t}^{*}}-V_{t}^{\pi_{K}}\|_{\sigma}$
$\displaystyle\leq\frac{2}{(1-\gamma)^{2}}\left\\{(1+\gamma)\sqrt{CC_{\sigma,\mu}^{\prime\prime}}\left[\frac{2}{\sqrt{1-\gamma^{2}}}\left(2\sqrt{2}E_{0}(\mathcal{F})+E_{2}\right)\right.\right.$
$\displaystyle\qquad\qquad\qquad\left.+\frac{2}{1-\gamma}\left(\gamma
V_{\text{max}}L\sqrt{\frac{d}{\nu_{\mu}}}(\sqrt{\frac{8\log(8dK/\delta)}{N}}+\frac{1}{N})\right)+E_{1}\right]$
$\displaystyle\qquad\qquad\qquad+\gamma^{\frac{K-1}{2}}R_{\text{max}}+3\epsilon\sqrt{2C_{\sigma,\mu}^{\prime}}\Bigg{\\}}.$

#### Proof:

For convenience, we will simply remove the task subscript whenever we refer to
variables associated with $\mathcal{M}$. Define

$\displaystyle d_{t}^{\pi}$ $\displaystyle=\mathcal{D}_{t}^{\pi}V^{\pi},$
$\displaystyle\tilde{d}_{t,k}$
$\displaystyle=\mathcal{D}_{t}^{\pi_{k}}\tilde{V}_{k-1},$ $\displaystyle
e_{k}$ $\displaystyle=\tilde{V}_{k}-\mathcal{T}^{\pi_{k}}\tilde{V}_{k},$
$\displaystyle E_{k}$ $\displaystyle=P^{\pi_{k+1}}(I-\gamma
P^{\pi_{k+1}})^{-1}-P^{\pi^{*}}(I-\gamma P^{\pi_{k}})^{-1},$ $\displaystyle
F_{k}$ $\displaystyle=P^{\pi_{k+1}}(I-\gamma
P^{\pi_{k+1}})^{-1}+P^{\pi^{*}}(I-\gamma P^{\pi_{k}})^{-1}.$

From the proof of Lemma 12 in [2], we get

$\displaystyle V^{\pi^{*}}-V^{\pi_{K}}\leq\gamma\sum_{k=0}^{K-1}(\gamma
P^{\pi^{*}})^{K-k-1}E_{k}e_{k}+(\gamma
P^{\pi^{*}})^{K}(V^{\pi^{*}}-V^{\pi_{0}}).$

By applying the above inequality, and taking the absolute value on both sides
point-wise, we get

$\displaystyle|V_{t}^{\pi_{t}^{*}}-V_{t}^{\pi_{K}}|$
$\displaystyle=|V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}}|+|V^{\pi^{*}}-V^{\pi_{K}}|+|V^{\pi_{K}}-V_{t}^{\pi_{K}}|$
$\displaystyle\leq\gamma\sum_{k=0}^{K-1}(\gamma
P^{\pi^{*}})^{K-k-1}F_{k}|e_{k}|+(\gamma
P^{\pi^{*}})^{K}|V^{\pi^{*}}-V^{\pi_{0}}|+|V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}}|+|V^{\pi_{K}}-V_{t}^{\pi_{K}}|$
$\displaystyle\leq\gamma\sum_{k=0}^{K-1}(\gamma
P^{\pi^{*}})^{K-k-1}F_{k}|e_{k}|+\frac{2R_{\text{max}}}{1-\gamma}\gamma^{K}+|V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}}|+|V^{\pi_{K}}-V_{t}^{\pi_{K}}|$

where we used the fact that
$|V^{\pi^{*}}-V^{\pi_{0}}|\leq(2R_{\text{max}}/(1-\gamma))\mathbf{1}$. Next,
we derive upper bounds for $|V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}}|$ and
$|V^{\pi_{K}}-V_{t}^{\pi_{K}}|$.

  * (a)

Observe that

$\displaystyle V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}}$
$\displaystyle=\mathcal{T}_{t}^{\pi_{t}^{*}}V_{t}^{\pi_{t}^{*}}-\mathcal{T}^{\pi^{*}}V^{\pi^{*}}$
$\displaystyle\leq\mathcal{T}_{t}^{\pi_{t}^{*}}V_{t}^{\pi_{t}^{*}}-\mathcal{T}^{\pi_{t}^{*}}V^{\pi^{*}}$
$\displaystyle=\mathcal{T}_{t}^{\pi_{t}^{*}}V_{t}^{\pi_{t}^{*}}-\mathcal{T}^{\pi_{t}^{*}}V_{t}^{\pi_{t}^{*}}+\mathcal{T}^{\pi_{t}^{*}}(V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}})$
$\displaystyle\leq(I-\gamma P_{t}^{\pi_{t}^{*}})^{-1}d_{t}^{\pi_{t}^{*}}.$

The first inequality follows from the fact that $\pi^{*}$ is optimal with
respect to $V^{\pi^{*}}$. The second inequality follows from the taylor
expansion of the inverse term. By closely following the same steps, we also
get

$\displaystyle V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}}$
$\displaystyle=\mathcal{T}_{t}^{\pi_{t}^{*}}V_{t}^{\pi_{t}^{*}}-\mathcal{T}^{\pi^{*}}V^{\pi^{*}}$
$\displaystyle\geq\mathcal{T}_{t}^{\pi^{*}}V_{t}^{\pi_{t}^{*}}-\mathcal{T}^{\pi^{*}}V^{\pi^{*}}$
$\displaystyle=\mathcal{T}_{t}^{\pi^{*}}V^{\pi^{*}}-\mathcal{T}^{\pi^{*}}V^{\pi^{*}}+\mathcal{T}_{t}^{\pi^{*}}(V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}})$
$\displaystyle\geq(I-\gamma P_{t}^{\pi^{*}})^{-1}d_{t}^{\pi^{*}}.$

By splitting into positive and negative components and applying the above
bounds, we get

$\displaystyle|V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}}|$
$\displaystyle=|(V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}})_{+}-(V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}})_{-}|$
$\displaystyle\leq|(V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}})_{+}|+|(V_{t}^{\pi_{t}^{*}}-V^{\pi^{*}})_{-}|$
$\displaystyle\leq|(I-\gamma
P_{t}^{\pi_{t}^{*}})^{-1}d_{t}^{\pi_{t}^{*}}|+|(I-\gamma
P_{t}^{\pi^{*}})^{-1}d_{t}^{\pi^{*}}|$ $\displaystyle\leq(I-\gamma
P_{t}^{\pi_{t}^{*}})^{-1}|d_{t}^{\pi_{t}^{*}}|+(I-\gamma
P_{t}^{\pi^{*}})^{-1}|d_{t}^{\pi^{*}}|$

  * (b)

Observe that

$\displaystyle V^{\pi_{K}}-V_{t}^{\pi_{K}}$
$\displaystyle\leq\mathcal{T}^{\pi_{K}}V^{\pi_{K}}+\mathcal{T}^{\pi_{K}}\tilde{V}_{K-1}-\mathcal{T}_{t}^{\pi_{K}}\tilde{V}_{K-1}-\mathcal{T}_{t}^{\pi_{K}}V_{t}^{\pi_{K}}$
$\displaystyle=\mathcal{T}^{\pi_{K}}V^{\pi_{K}}+\mathcal{T}^{\pi_{K}}\tilde{V}_{K-1}-\mathcal{T}_{t}^{\pi_{K}}\tilde{V}_{K-1}-\mathcal{T}_{t}^{\pi_{K}}V^{\pi_{K}}+\mathcal{T}_{t}^{\pi_{K}}(V^{\pi_{K}}-V_{t}^{\pi_{K}})$
$\displaystyle\leq(I-\gamma
P_{t}^{\pi_{K}})^{-1}(-d_{t}^{\pi_{K}}-\tilde{d}_{t,K}).$

The first inequality follows from the fact that $\pi_{K}$ is optimal with
respect to $\tilde{V}_{K-1}$. The second inequality follows from the taylor
expansion of the inverse term. By closely following the same steps, we also
get

$\displaystyle V^{\pi_{K}}-V_{t}^{\pi_{K}}$
$\displaystyle\geq\mathcal{T}^{\pi_{K}}V^{\pi_{K}}-\mathcal{T}^{\pi_{K}}\tilde{V}_{K-1}+\mathcal{T}_{t}^{\pi_{K}}\tilde{V}_{K-1}-\mathcal{T}_{t}^{\pi_{K}}V_{t}^{\pi_{K}}$
$\displaystyle=\mathcal{T}^{\pi_{K}}V^{\pi_{K}}-\mathcal{T}^{\pi_{K}}\tilde{V}_{K-1}+\mathcal{T}_{t}^{\pi_{K}}\tilde{V}_{K-1}-\mathcal{T}_{t}^{\pi_{K}}V^{\pi_{K}}+\mathcal{T}_{t}^{\pi_{K}}(V^{\pi_{K}}-V_{t}^{\pi_{K}})$
$\displaystyle\geq(I-\gamma
P_{t}^{\pi_{K}})^{-1}(-d_{t}^{\pi_{K}}+\tilde{d}_{t,K}).$

By splitting into positive and negative components and applying the above
bounds, we get

$\displaystyle|V^{\pi_{K}}-V_{t}^{\pi_{K}}|$
$\displaystyle=|(V^{\pi_{K}}-V_{t}^{\pi_{K}})_{+}-(V^{\pi_{K}}-V_{t}^{\pi_{K}})_{-}|$
$\displaystyle\leq|(V^{\pi_{K}}-V_{t}^{\pi_{K}})_{+}|+|(V^{\pi_{K}}-V_{t}^{\pi_{K}})_{-}|$
$\displaystyle=|(I-\gamma
P_{t}^{\pi_{K}})^{-1}(-d_{t}^{\pi_{K}}-\tilde{d}_{t,K})|+|(I-\gamma
P_{t}^{\pi_{K}})^{-1}(-d_{t}^{\pi_{K}}+\tilde{d}_{t,K})|$
$\displaystyle\leq(I-\gamma
P_{t}^{\pi_{K}})^{-1}|-d_{t}^{\pi_{K}}-\tilde{d}_{t,K}|+(I-\gamma
P_{t}^{\pi_{K}})^{-1}|-d_{t}^{\pi_{K}}+\tilde{d}_{t,K}|$ $\displaystyle\leq
2(I-\gamma P_{t}^{\pi_{K}})^{-1}(|d_{t}^{\pi_{K}}|+|\tilde{d}_{t,K}|).$

By applying the upper bounds from (a) and (b), we get

$\displaystyle|V_{t}^{\pi_{t}^{*}}-V_{t}^{\pi_{K}}|$
$\displaystyle\leq\frac{2(1-\gamma^{K+2})}{(1-\gamma)^{2}}\left[\sum_{k=0}^{K-1}\alpha_{k}A_{k}|e_{k}|+\alpha(R_{\text{max}}/\gamma)\right.$
$\displaystyle\qquad\qquad\qquad\qquad+(\beta/6)B^{\pi_{t}^{*}}\cdot
6|d_{t}^{\pi_{t}^{*}}|+(\beta/6)B^{\pi^{*}}\cdot 6|d_{t}^{\pi^{*}}|$
$\displaystyle\qquad\qquad\qquad\qquad+(\beta/3)B^{\pi_{K}}\cdot
6|d_{t}^{\pi_{K}}|+(\beta/3)B^{\pi_{K}}\cdot 6|\tilde{d}_{t,K}|\Bigg{]}$

where we introduced the positive coefficients

$\displaystyle\alpha_{k}$
$\displaystyle=\frac{(1-\gamma)}{1-\gamma^{K+2}}\gamma^{K-k},\quad\text{for}\;0\leq
k<K,$ $\displaystyle\alpha$
$\displaystyle=\frac{(1-\gamma)}{1-\gamma^{K+2}}\gamma^{K+1},$
$\displaystyle\beta$ $\displaystyle=\frac{(1-\gamma)}{2(1-\gamma^{K+2})},$

and the operators

$\displaystyle A_{k}$
$\displaystyle=\frac{1-\gamma}{2}(P^{\pi^{*}})^{K-k-1}F_{k},\quad\text{for}\;0\leq
k<K,$ $\displaystyle B^{\pi}$ $\displaystyle=(1-\gamma)(I-\gamma
P_{t}^{\pi})^{-1}.$

Let $\lambda_{K}=\left[\frac{2(1-\gamma^{K+2})}{(1-\gamma)^{2}}\right]^{p}$.
Note that the coefficients $\alpha_{k}$, $\alpha$, and $\beta$, sum to $1$,
and the operators are positive linear operators that satisfy
$A_{k}\mathbf{1}=\mathbf{1}$ and $B^{\pi}\mathbf{1}=\mathbf{1}$. Therefore, by
taking the $p$th power on both sides, applying Jensen’s inequality twice, and
then integrating both sides with respect to $\sigma(x)$, we get

$\displaystyle\|V_{t}^{\pi_{t}^{*}}-V_{t}^{\pi_{K}}\|_{p,\sigma}^{p}$
$\displaystyle=\int\sigma(dx)|V_{t}^{\pi_{t}^{*}}-V_{t}^{\pi_{K}}|^{p}$
$\displaystyle\leq\lambda_{K}\sigma\left[\sum_{k=0}^{K-1}\alpha_{k}A_{k}|e_{k}|^{p}+\alpha(R_{\text{max}}/\gamma)^{p}\right.$
$\displaystyle\qquad\qquad+(\beta/6)B^{\pi_{t}^{*}}(6|d_{t}^{\pi_{t}^{*}}|)^{p}+(\beta/6)B^{\pi^{*}}(6|d_{t}^{\pi^{*}}|)^{p}$
$\displaystyle\qquad\qquad+(\beta/3)B^{\pi_{K}}(6|d_{t}^{\pi_{K}}|)^{p}+(\beta/3)B^{\pi_{K}}(6|\tilde{d}_{t,K}|)^{p}\Bigg{]}.$

From the definition of the coefficients $c_{\sigma,\mu}(m)$, we get

$\displaystyle\sigma A_{k}$ $\displaystyle\leq(1-\gamma)\sum_{m\geq
0}\gamma^{m}c_{\sigma,\mu}(m+K-k)\mu,$ $\displaystyle\sigma B^{\pi}$
$\displaystyle\leq(1-\gamma)\sum_{m\geq 0}\gamma^{m}c_{\sigma,\mu}(m)\mu.$

Therefore, it follows that

$\displaystyle\sigma\left[\sum_{k=0}^{K-1}\alpha_{k}A_{k}|e_{k}|^{p}\right]$
$\displaystyle\leq(1-\gamma)\sum_{k=0}^{K-1}\alpha_{k}\sum_{m\geq
0}\gamma^{m}c_{\sigma,\mu}(m+K-k)\mu|e_{k}|^{p}$
$\displaystyle=\frac{\gamma(1-\gamma)^{2}}{1-\gamma^{K+2}}\sum_{k=0}^{K-1}\sum_{m\geq
0}\gamma^{m+K-k-1}c_{\sigma,\mu}(m+K-k)\|e_{k}\|_{p,\mu}^{p}$
$\displaystyle\leq\frac{\gamma}{1-\gamma^{K+2}}C_{\sigma,\mu}^{\prime\prime}e^{p}$

where $e=\max_{0\leq k<K}\|e_{k}\|_{p,\mu}^{p}$. The terms involving $B^{\pi}$
satisfy

$\displaystyle\sigma\left[B^{\pi}(6|d_{t}^{\pi}|)^{p}\right]$
$\displaystyle\leq 6^{p}(1-\gamma)\sum_{m\geq
0}\gamma^{m}c_{\sigma,\mu}(m)\mu|d_{t}^{\pi}|^{p}\leq
6^{p}C_{\sigma,\mu}^{\prime}\|d_{t}^{\pi}\|_{p,\mu}^{p}.$

Putting all these together, and choosing $p=2$, we get

$\displaystyle\|V_{t}^{\pi_{t}^{*}}-V_{t}^{\pi_{K}}\|_{\sigma}$
$\displaystyle\leq\lambda_{K}^{\frac{1}{2}}\left[\frac{\gamma}{1-\gamma^{K+2}}C_{\sigma,\mu}^{\prime\prime}e^{2}+\frac{(1-\gamma)\gamma^{K+1}}{1-\gamma^{K+2}}(R_{\text{max}}/\gamma)^{2}+\frac{36(1-\gamma)}{2(1-\gamma^{K+2})}C_{\sigma,\mu}^{\prime}\epsilon^{2}\right]^{\frac{1}{2}}$
$\displaystyle\leq\frac{2}{(1-\gamma)^{2}}\left[\gamma
C_{\sigma,\mu}^{\prime\prime}e^{2}+(1-\gamma)\gamma^{K+1}(R_{\text{max}}/\gamma)^{2}+\frac{36(1-\gamma)}{2}C_{\sigma,\mu}^{\prime}\epsilon^{2}\right]^{\frac{1}{2}}$
$\displaystyle\leq\frac{2}{(1-\gamma)^{2}}\left[C_{\sigma,\mu}^{\prime\prime}e^{2}+\gamma^{K+1}(R_{\text{max}}/\gamma)^{2}+18C_{\sigma,\mu}^{\prime}\epsilon^{2}\right]^{\frac{1}{2}}$
$\displaystyle\leq\frac{2}{(1-\gamma)^{2}}\left[\sqrt{C_{\sigma,\mu}^{\prime\prime}}e+\gamma^{\frac{K-1}{2}}R_{\text{max}}+3\epsilon\sqrt{2C_{\sigma,\mu}^{\prime}}\right].$

The desired result can then be obtained by applying the same steps as in the
proof of Theorem 8 in [14].

$\square$

###  7.1 Financial Portfolio Optimization: Additional Details

The dataset consists of daily prices for $68$ instruments in the technology
and communication sectors from 2009 to 2019. We use 2009–2018 for training and
2019 for testing. To validate that our approach learns common features across
instruments, and thus can transfer, we reserve $18$ instruments not seen
during training for further testing. The global asset universe $\mathcal{U}$
used for training contains $50$ instruments.

We construct tasks by randomly choosing a portfolio of $|\mathcal{U}_{t}|=10$
instruments for each task. We create a permutation invariant policy network by
applying the same sequence of operations to every instrument state. That is,
for each instrument, the flattened input prices are passed through a common
RNN with 25 hidden units and tanh activation, this output is concatenated with
the latest allocation fraction of the instrument, and passed through a common
dense layer to produce a score. Instrument scores are passed to a softmax
function to produce allocations that sum to one. The smoothing parameter for
the scores $\gamma=0.2$, $\alpha=0.5$ for the task prioritisation parameter
and $\beta=1.0$ to fully compensate for the prioritized sampling bias.

###  7.2 Meta Federated Learning

Suppose we have a universe of federated learning clients $\mathcal{U}$. The
goal of task $t$ is to aggregate models in a federated learning experiment
over a subset of clients $\mathcal{U}_{t}\subseteq\mathcal{U}$. At each step
$n$, the action $a_{i,n}$ represents the weight assigned to the supervised
learning model of client $i$ in the averaging procedure. Let $v_{i,n}$ denote
the model of the client (i.e. the tensor of model parameters). We model the
state of the client as some function of its $H$ most recent models
$x_{i,n}=f(v_{i,n-H+1},\ldots,v_{i,n})$. Assume that the aggregator has access
to a small evaluation dataset that it can use to approximately assess the
quality of models. We define the reward at each step to be the accuracy of the
aggregate model,
$R_{t}(x_{n},a_{n})=\mathcal{L}\left(\sum_{i\in\mathcal{U}_{t}}a_{i,n}v_{i,n}\right)$,
where $\mathcal{L}(v)$ is a function that provides the accuracy of a model $v$
on the evaluation dataset. Therefore, by maximizing the total return over all
time periods, we seek to maximize both the accuracy at the final time step as
well as the time to convergence. We optimize the policy using Proximal Policy
Optimization (PPO).

We use the MNIST digit recognition problem. Each client observes $600$ samples
from the train dataset and trains a classifier composed of one $5$x$5$
convolutional layer (with $32$ channels and ReLu activation) and a softmax
output layer. We use the same permutation invariant policy network
architecture as before with $10$ hidden units in the RNN. We randomly select
$|\mathcal{U}_{t}|=10$ clients for each task. We learn using an evaluation
dataset comprised of $1000$ random samples from the test dataset and test
using all $10000$ samples in the test dataset. We fix the number of federated
learning iterations to $50$.

We explore the benefit of MTL in identifying useful clients in scenarios with
skewed data distribution. We partition the dataset such that $8$ of the
clients in each task observe random digits between $0$ to $5$ and the
remaining $2$ clients observe random digits between $6$ to $9$. Therefore, for
each task, $20\%$ of the clients possess $40\%$ of the unique labels. The
state of each client are the accuracies of its $H$ most recent models on the
evaluation dataset.

Figure 6 shows the potential benefits of multi-task learning when simulators
are inaccurate. In particular, we obtain two aggregation policies, one trained
using single-task learning (STL), and another trained using multi-task
learning (MTL), both trained using the same number of steps, and we observe
their behavior during testing. The plots show that multi-task learning is able
to learn non-uniform averaging policies that improve the convergence and
performance of federated learning runs. More importantly, it can perform
better than single-task learning even with the same number of samples. This
may be attributed to the wider variety of client configurations (and
consequently experiences) in the multi-task approach.

Figure 6:  These plots compare the behavior of a multi-task policy and a
single-task policy during testing. FedAvg denotes the accuracy of federated
learning with uniform averaging. The left plot shows the accuracy of the
aggregate model during federated learning. The right two plots show the
weights produced by the policy for different clients. Note that clients $8$
and $9$ possess $40\%$ of the unique labels.

## References

  * [1] Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio.  Variance reduction in sgd by distributed importance sampling, 2015.
  * [2] András Antos, Csaba Szepesvári, and Rémi Munos.  Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path.  Machine Learning, 71(1):89–129, 2008.
  * [3] Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause.  Safe model-based reinforcement learning with stability guarantees.  In NIPS, 2017.
  * [4] Timo Bram, Gino Brunner, Oliver Richter, and Roger Wattenhofer.  Attentive multi-task deep reinforcement learning.  In ECML/PKDD, 2019.
  * [5] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine.  Data-efficient model-based reinforcement learning with deep probabilistic dynamics models.  In NIPS 2018, 2018.
  * [6] Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel.  Model-based reinforcement learning via meta-policy optimization.  ArXiv, abs/1809.05214, 2018.
  * [7] Carlo D’Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters.  Sharing knowledge in multi-task deep reinforcement learning.  In International Conference on Learning Representations, 2020.
  * [8] Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt.  Multi-task deep reinforcement learning with popart.  In AAAI, 2018.
  * [9] Wenzhen Huang, Junge Zhang, and Kaiqi Huang.  Bootstrap estimated uncertainty of the environment model for model-based reinforcement learning.  In AAAI, 2019.
  * [10] Zhengyao Jiang, Dixing Xu, and Jinjun Liang.  A deep reinforcement learning framework for the financial portfolio management problem, 2017.
  * [11] Angelos Katharopoulos and Francois Fleuret.  Not all samples are created equal: Deep learning with importance sampling.  In ICML, 2018.
  * [12] Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, D. Parikh, and Dhruv Batra.  Modeling the long term future in model-based reinforcement learning.  In ICLR, 2019.
  * [13] Alessandro Lazaric and Mohammad Ghavamzadeh.  Bayesian multi-task reinforcement learning.  In ICML, 2010.
  * [14] Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos.  Finite-sample analysis of least-squares policy iteration.  Journal of Machine Learning Research, 13(Oct):3041–3074, 2012.
  * [15] Iou-Jen Liu, Raymond A. Yeh, and Alexander G. Schwing.  PIC: permutation invariant critic for multi-agent deep reinforcement learning.  In 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings, pages 590–602, 2019.
  * [16] Ilya Loshchilov and Frank Hutter.  Online batch selection for faster training of neural networks, 2015.
  * [17] Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov.  Actor-mimic: Deep multitask and transfer reinforcement learning.  CoRR, abs/1511.06342, 2015.
  * [18] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  ArXiv, abs/1606.04671, 2016.
  * [19] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  In International Conference on Learning Representations, 2016.
  * [20] Iulian Serban, Chinnadhurai Sankar, Michael Pieper, Joelle Pineau, and Yoshua Bengio.  The bottleneck simulator: A model-based deep reinforcement learning approach.  ArXiv, abs/1807.04723, 2018.
  * [21] Yee Whye Teh, Victor Bapst, Wojciech Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Manfred Otto Heess, and Razvan Pascanu.  Distral: Robust multitask reinforcement learning.  In NIPS, 2017.
  * [22] Hado van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver.  Learning values across many orders of magnitude.  In NIPS, 2016.
  * [23] Tung-Long Vuong, Do Van Nguyen, Tai-Long Nguyen, Cong-Minh Bui, Hai-Dang Kieu, Viet-Cuong Ta, Quoc-Long Tran, and Thanh Ha Lê.  Sharing experience in multitask reinforcement learning.  In IJCAI, 2019.
  * [24] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli.  Multi-task reinforcement learning: a hierarchical bayesian approach.  In ICML ’07, 2007.
  * [25] Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma.  Algorithmic framework for model-based reinforcement learning with theoretical guarantees.  ArXiv, abs/1807.03858, 2019.
  * [26] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus.  Improving sample efficiency in model-free reinforcement learning from images.  ArXiv, abs/1910.01741, 2019.
  * [27] Yang Yu.  Towards sample efficient reinforcement learning.  In IJCAI, 2018.
  * [28] Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine.  Solar: Deep structured latent representations for model-based reinforcement learning.  ArXiv, abs/1808.09105, 2018.


capbtabboxtable[][]

# Autonomous robotic re-alignment for
face-to-face underwater human-robot interaction††thanks: This work was
supported in part by the Science, Mathematics, and Research for Transformation
(SMART) Scholarship provided through the US Department of Defense.

Demetrious T. Kutzke1,†, Ashwin Wariar2,†, and Junaed Sattar3, The authors are
with the Department of Computer Science & Engineering and the Minnesota
Robotics Institute, University of Minnesota–Twin Cities, Minneapolis, MN
55455, USA {1kutzk015, 2waria012<EMAIL_ADDRESS>
(November 26, 2023)

###### Abstract

The use of autonomous underwater vehicles (AUVs) to accomplish traditionally
challenging and dangerous tasks has proliferated thanks to advances in
sensing, navigation, manipulation, and on-board computing technologies.
Utilizing AUVs in underwater human-robot interaction (UHRI) has witnessed
comparatively smaller levels of growth due to limitations in bi-directional
communication and significant technical hurdles to bridge the gap between
analogies with terrestrial interaction strategies and those that are possible
in the underwater domain. A necessary component to support UHRI is
establishing a system for safe robotic-diver approach to establish face-to-
face communication that considers non-standard human body pose. In this work,
we introduce a stereo vision system for enhancing UHRI that utilizes three-
dimensional reconstruction from stereo image pairs and machine learning for
localizing human joint estimates. We then establish a convention for a
coordinate system that encodes the direction the human is facing with respect
to the camera coordinate frame. This allows automatic setpoint computation
that preserves human body scale and can be used as input to an image-based
visual servo control scheme. We show that our setpoint computations tend to
agree both quantitatively and qualitatively with experimental setpoint
baselines. The methodology introduced shows promise for enhancing UHRI by
improving robotic perception of human orientation underwater.

##  1 Introduction

Figure 1: Underwater human robot interaction is enhanced by the robot’s
ability to re-orient itself with respect to the diver, rather than requiring
the diver to re-orient with respect to the robot.

Problems associated with humans and robots interacting, also referred to as
human-robot interaction (HRI), is well-studied in controlled terrestrial
environments [1]. Innovations in HRI have bolstered adoption of these
technologies into many areas of life, such as manufacturing [2], medicine [3],
long-term care of the elderly [4], military applications [5], and the
underwater domain [6]. This is due in part to the benefit of allowing robots
to take on the dirty, dull, and dangerous tasks [7] that would otherwise place
humans in direct harm or assist in situations in which it is not possible for
the human to provide the level of persistent attention required, as in the
case of long-term care facilities. The thought goes that off-loading these
tasks to robots will allow humans to interact with them from relative safety
or convenience, while also performing oversight [8].

Underwater human-robot interaction (UHRI) is much more challenging, because
the underwater domain presents a formidable environment for robotic sensing.
It lacks many of the benefits of the terrestrial domain such as high bandwidth
radio communication, _e.g._ , wireless internet and Bluetooth, consistent
lighting conditions, and localization via global-positioning satellites.
Underwater robots must utilize alternative methods to perform the same tasks
as their terrestrial counterparts. Often, expensive sonar-based techniques are
used to perform navigation and localization [9]. Utilizing visual sensors is
also challenging, because differences in salinity and particulates in the
water can occlude and distort imagery. Creative techniques must be employed to
enhance vision underwater when the conditions are especially degraded [10, 11,
12]. However, even with the challenges of visual imagery, there are instances
where utilizing camera data is preferable to expensive and invasive acoustic
systems. For example, when acoustically susceptible marine life such as
dolphins or whales are present [13], in these instances, utilizing visual
sensing, which is less invasive, is both ethical and beneficial to the
preservation of the marine life. Divers operate in a similarly sensory-
deprived state. Scuba masks occlude peripheral vision or at the very least can
reduce the diver’s ability to see or perceive dynamic robotic gestures
underwater; acoustic signals are degraded by inhalation and exhalation through
the breathing regulator, which significantly reduces the diver’s ability to
hear; and environmental conditions such as strong currents, silting from
sediment in the water column, and frigid water temperatures all contribute to
generally high cognitive loads. Sensory-deprived states for both robots and
humans means that in complex UHRI scenarios, where communication is critical
from both robot-to-human (R2H) and human-to-robot (H2R), there is a high-
probability of information loss. We argue that because of these conditions,
the robot must have the ability to autonomously establish face-to-face (F2F)
communication. F2F communication reduces the probability of information loss
by ensuring that the robot and the human are within a safe distance and in
full view of each other. The diver can see the robot’s movements and vice-
versa. The diver also has the best chance of hearing any acoustically
communicated information.

To achieve this F2F configuration, we propose a stereo vision algorithm to
autonomously compute a desired feature setpoint, which can be used for visual
servo control schemes. This eliminates the need for human-engineered features
and equips the robot with the ability to infer a desired F2F setpoint from
nonstandard body poses that preserves scale. Scale is important to ensure safe
and consistent approach distances for divers of different shapes. To our
knowledge, the problem of autonomously establishing F2F communication
underwater has not been considered for general body poses or instances where
the robot is not already within a safe communication distance and can perceive
the human diver’s face.

Many techniques have been devised to establish both R2H [14] and H2R
communication (_e.g._ , [15]). These systems are supported by complementary
techniques to enable visual robot control to place the robot within a safe
distance of the human [16]. This allows higher fidelity understanding from
both the robot’s and the human’s perspective, since it is thought that the
information exchange is best when interpreted from the alignment between the
human’s eyes and the robot’s camera [16]. To ensure robust communication,
authors in [17] used a transformer-driven network for detecting diver gaze
based on facial mask keypoints. However, their work does not handle general
poses Fig. 2, or those in which the facial keypoints are not visible.

We argue that a complementary problem to the works of [16] and [17] is
utilizing a two-dimensional pose estimator, along with a stereo visual
approach to establish three-dimensional positions. By doing this, we can
accommodate non-standard human poses, such as those shown in Fig. 2. This will
ultimately enable more complex robotic control for re-orientation; _e.g._ ,
when the human is conducting complex tasks and is unable to re-orient
themselves with respect to the robot, the robot can come to the human. It is
from this perspective that we define our primary contributions to support
UHRI, which can be summarized as follows:

  * •

The aggregation of a diverse torso keypoint dataset and results from training
an off-the-shelf pose estimation algorithm for two-dimensional human pose
estimates that accommodates non-standard body poses.

  * •

The computation of an alignment vector and establishment of a convention for
assigning a coordinate frame to a human’s facing direction.

  * •

Scale preserving setpoint computations which preserve different body shapes at
different distances between the robot and the human.

(a)

(b)

(c)

(d)

Figure 2: Example non-standard diver poses that are typical during scuba
diving operations. Diver robot interaction scenarios must accommodate these
poses to be useful for underwater missions.

##  2 Related Work

The work introduced in this paper exists within the boundary between UHRI and
computer vision for pose estimation. Here we discuss some of the works that
influence our methodology.

Underwater human robot interaction. Various methods have been proposed for
robotic detection and individual identification of human divers (_e.g._ , [18,
19, 20]) with features extracted from visual, or spatio-temporal signals.
Others have utilized sonar detection mechanisms to both directly detect in
frequency space the presence of a diver [21] and reconstructed acoustic images
[22, 23]. For explicit communication between an AUV and divers, both robot-to-
human and human-to-robot methods have been proposed; _e.g._ , robots have used
light [14], motion [6, 24], and other cues to communicate intent and
information to divers, and fiducials [25, 26, 27], hand gestures [28, 29, 30,
31], and complex user interface devices [32, 33] have all been used by divers
to control robots. However, it is conceivably challenging and constraining for
divers to use tags or UI devices while underwater for certain tasks.

Human Pose Estimation. Pose estimation is the task of determining a set of
keypoints that define human joint positions in an image. Various techniques
exist, but most rely on convolutional neural networks (CNNs) [34] to perform
feature extraction and output heatmaps over candidate locations [35, 36, 37].

The networks are trained to regress from heatmaps to perform keypoint
localization by selecting the location with the highest probability as the
most likely joint location.

Localizing joint locations accurately is a significant challenge underwater,
which is exacerbated by inconsistent lighting conditions and the lack of
saliency, or pronounced features, within the typical diver silhouette. Chavez
et al. utilize a recurrent neural network (RNN) with long short-term (LSTM)
cells to learn the sequential joint orientations affixed to the human diver,
exploiting stereo vision and $17$ inertial measurement units (IMUs) that
communicate the diver’s movements acoustically. We recognize that placing
additional burden on the diver’s already intense cognitive load is
problematic. Instead, we endeavor to construct F2F re-orientation in such a
way that the robot re-orients itself with respect to the human based off of
image observations alone.

##  3 The Face-to-Face Reorientation Approach

The F2F scale-preserving setpoint computation comprises two components. First,
a pose estimation component localizes torso keypoints, and second, an
alignment vector computation establishes a convention for affixing a right-
handed coordinate system to the human, from which we compute the
transformation that anti-aligns the body frame coordinate system with the
camera frame. Perspective projection allows us to recover the ideal setpoint,
which is the configuration in which the human is facing the camera.

While a future goal is to use a three-dimensional pose estimation algorithm on
monocular camera data, much of the work in three-dimensional pose estimation
first uses multi-camera setups to triangulate pose keypoints to provide a
z-component to ground truth labeled data. Pose estimation algorithms can then
be trained to directly predict a three-dimensional vector from a single
monocular image, see [38, 39], for example. In the underwater domain,
instrumenting an experimental setup with calibrated multi-view cameras is a
challenge and impractical for most setups. To that end, we utilize calibrated
stereo cameras affixed to a robot to collect and aggregate images of human
divers. We trained a deep neural network on labeled image data to localize
two-dimensional human joints. During runtime, we utilize stereo reconstruction
based on pose alone. Dense stereo matching is found to be ineffective for the
underwater environment. The typical diver silhouette almost entirely appears
the same to traditional block matching algorithms or even more sophisticated
post-processing techniques that fill in gaps in the disparity map. From the
scale-preserving setpoint computation, we preserve different body shapes to
ensure that the robot can automatically predict the optimal alignment or
setpoint for the visual control scheme, without need for human intervention or
calibration before the beginning of the mission. Instead, the robot can detect
the setpoint using our method and perform classic visual control schemes. A
detailed account of visual servo control schemes is beyond the scope of the
present work, but an extensive treatment can be found in [40, 41, 42]. Stereo
Diver Pose Dataset. We aggregated and labeled $6,711$ stereo image pairs from
a closed-water environment in which a stereo vision camera was used to collect
images of divers in diverse poses. The images collected are all of size
$640\times 480$ pixels, and feature a single diver in a full wet suit and dive
gear. The hands and face are exposed, except for parts of the face, which are
partially obscured by the breathing regulator and mask. The poses were labeled
according to the convention shown in Fig. 3. We argue that the points that
encompass the torso, the midpoint of the eyes, and the base of the neck, are
the primary keypoints for robotic re-orientation. The torso keypoints do not
change relative to each other throughout normal diving operations, whereas the
limbs are often moving to stabilize the diver’s position in the water. The
midpoint of the eyes was also utilized to accurately reflect the midpoint of
the body itself. Without this additional keypoint, the robot would come face-
to-face lower than expected with the diver, inhibiting the most efficient
communication. Example poses are shown in Fig. 4.

Figure 3: Pose keypoint convention used by F2F along with a sample labeled
image. The pose estimator may provide more anatomical keypoints, but F2F only
requires these six.

(a)

(b)

(c)

Figure 4: Sample raw images from the F2F pose dataset. The dataset contains
diverse poses to represent the broadest possible set of orientations a diver
can assume while conducting underwater operations.

Pose Estimation. With the aggregated dataset, we train a deep neural network
based on the DeepLabCut (DLC) framework [43, 44] for torso keypoint
estimation. Specifically, we train a ResNet-50-based neural network using
$95$% of the dataset, or approximately $6,375$ images, for $500,000$ training
iterations. We find the test error to be $12.75$ pixels, and the train error
at $11.14$ pixels. We then use a threshold p-cutoff of $0.05$ to condition the
(x,y) coordinates for future analysis. While the error might seem high, our
goal is to quantify the extent to which we consider alignment as face-to-face
interaction. To that end, we do not need nearly-optimal keypoint
configurations, but localizations that are good enough to recover sparse 3D
reconstructions are sufficient; we find that approximately $10$ pixels of
error provides enough accuracy for our application.

Examples from the pose estimator evaluated on the test dataset are shown in
Fig. 5 below. Generally, DLC performs adequately localizing the joints.
However, there are times where even variations in labeling likely contribute
to issues in localization. For example, if the joint is occluded, it is
possible the DLC network will be unable to predict location.

(a)

(b)

Figure 5: DeepLabCut evaluation on the F2F test dataset. Cross markers
indicate ground truth labels, and dots indicate DeepLabCut estimations with
confidences $p>p_{\text{cutoff}}=0.05$.

Alignment Coordinate System Convention. A key contribution of this work is the
computation of the alignment vector. The alignment vector extends
perpendicularly to the plane defined by the torso of the diver. This serves as
the anchor vector, which allows us to define a coordinate system that is
attached to the diver. To construct this vector, we use a set of joint
location keypoints from our pose estimation network. Let $\vec{k}_{\text{b}}$
define the bridge of the nose, $\vec{k}_{\text{n}}$ define the base of the
neck, $\vec{k}_{\text{rs}}$ define the right shoulder, $\vec{k}_{\text{rh}}$
define the right hip, $\vec{k}_{\text{lh}}$ define the left hip, and
$\vec{k}_{\text{ls}}$ define the left shoulder.

After performing stereo image rectification and triangulating based on joint
localizations from both the left and right image pairs, the vectors
$\vec{k}_{i}\in\mathcal{K}$, where $i$ corresponds to a specific joint, are
defined in three-dimensional camera coordinates as
$({}^{c}x_{i},{}^{c}y_{i},{}^{c}z_{i})$. Let $\vec{s}_{i}$ denote the
corresponding projection of the point $\vec{k}_{i}$ onto the image plane of
the robot’s camera. The projection $\vec{s}_{i}=(u_{i},v_{i})$ is a point
defined in the image plane, where $u_{i}\in[0,N]$ and $v_{i}\in[0,M]$, and
$N,M$ are the image width and height, respectively. To compute the alignment
vector, we define the following steps.

  1. 1.

We compute the center of the predicted keypoints as
$\vec{k}_{\text{o}}=\langle\vec{k}\rangle_{\vec{k}\in\mathcal{K}}$, where
$\langle\cdot\rangle$ defines the vector average computation.

The resultant vector $\vec{k}_{\text{o}}$ is located approximately center of
mass and is skewed toward the upper part of the torso.

  2. 2.

We define several difference vector quantities that exist on the torso plane
as

$\displaystyle\vec{k}_{\text{lsh}}$
$\displaystyle=\vec{k}_{\text{ls}}-\vec{k}_{\text{lh}}$
$\displaystyle\vec{k}_{\text{nlh}}$
$\displaystyle=\vec{k}_{\text{n}}-\vec{k}_{\text{lh}}$ (1)
$\displaystyle\vec{k}_{\text{nrh}}$
$\displaystyle=\vec{k}_{\text{n}}-\vec{k}_{\text{rh}}$
$\displaystyle\vec{k}_{\text{rsh}}$
$\displaystyle=\vec{k}_{\text{rs}}-\vec{k}_{\text{rh}}.$ (2)

These quantities are needed to establish the relationships between joint
locations, effectively defining the torso plane and conditioning the
proceeding analysis with respect to the torso plane.

  3. 3.

We compute the alignment direction by taking the average direction of the
cross product between the difference vectors of the torso and neck joints.
This defines a direction perpendicular to the plane defined by the torso
keypoints

$\displaystyle\vec{k}_{\text{l}_{\times}}$
$\displaystyle=\vec{k}_{\text{lsh}}\times\vec{k}_{\text{nlh}}$ (3)
$\displaystyle\vec{k}_{\text{r}_{\times}}$
$\displaystyle=\vec{k}_{\text{nrh}}\times\vec{k}_{\text{rsh}}.$ (4)

To compute the average direction and define a unit vector, we first take the
average, and then we divide by the vector $L2$-norm

${}^{c}\hat{z}_{B}\equiv\frac{\langle\vec{k}_{\text{r}_{\times}},\vec{k}_{\text{l}_{\times}}\rangle}{\lVert\langle\vec{k}_{\text{r}_{\times}},\vec{k}_{\text{l}_{\times}}\rangle\rVert_{2}}.$
(5)

The alignment vector given by (5) points in a direction perpendicular to the
plane defined by the torso keypoints. We now affix a right-handed coordinate
system to $\vec{k}_{\text{o}}$, with ${}^{c}\hat{z}_{B}$ aligned along the
direction given in (5).

We choose ${}^{c}\hat{y}_{B}$ to be the vector that points along the direction
of the midpoint between hip joints. This is given by computing the midpoint of
the line segment connecting the hip joints

$\vec{k}_{\text{midpt}}=\langle\vec{k}_{\text{lh}},\vec{k}_{\text{rh}}\rangle.$
(6)

  4. 4.

From this we compute the unit vector that points from the center of mass
vector $\vec{k}_{o}$ to $\vec{k}_{\text{midpt}}$. This unit vector is defined
to be ${}^{c}\hat{y}_{B}$

${}^{c}\hat{y}_{B}=\frac{\vec{k}_{\text{midpt}}-\vec{k}_{o}}{\lVert\vec{k}_{\text{midpt}}-\vec{k}_{o}\rVert_{2}}.$
(7)

  5. 5.

Finally, the ${}^{c}\hat{x}_{B}$ is computed through a cross product
${}^{c}\hat{x}_{B}={}^{c}\hat{y}_{B}\times{}^{c}\hat{z}_{B}$. Together these
constitute the body frame
${}^{c}\mathcal{F}_{B}=[{}^{c}\hat{x}_{B},{}^{c}\hat{y}_{B},{}^{c}\hat{z}_{B},\vec{k}_{\text{o}}]$
of the human diver, affixed to the midpoint of the extracted pose keypoints,
with the ${}^{c}\hat{z}_{B}$ aligned in the direction perpendicular to the
plane defined by the torso keypoints. It is from this that we can then compute
the ideal pose configuration by anti-aligning the body frame with the camera
frame unit vectors.

The preceding analysis computes the body frame ${}^{c}\mathcal{F}_{B}$ with
respect to the camera frame. To compute the pose setpoint for a visual control
scheme, we need to compute the transformation that anti-aligns the camera
frame and the body frame. That is, there exists some transformation
$\tilde{\text{T}}$ that yields an ideal configuration of the body frame
${}^{c}\mathcal{F}^{*}_{B}$

${}^{c}\mathcal{F}^{*}_{B}=\tilde{\text{T}}{}^{c}\mathcal{F}_{B}.$ (8)

This transformation is not known a priori.

For a camera that has z-axis along the optical axis, a right-facing x-axis,
and a down-facing y-axis, the following constraints hold

$\displaystyle{}^{c}\hat{z}^{*}_{B}\cdot{}^{c}\hat{z}$ $\displaystyle=-1$
$\displaystyle{}^{c}\hat{x}^{*}_{B}\cdot{}^{c}\hat{x}$ $\displaystyle=-1$
$\displaystyle{}^{c}\hat{y}^{*}_{B}\cdot{}^{c}\hat{y}$ $\displaystyle=1.$ (9)

From these equations, we compute the rotation matrix required to align the
body frame with respect to the camera frame to be in an F2F orientation as

${}^{c}\mathcal{F}^{*}_{B}=\tilde{\text{T}}{}^{c}\mathcal{F}_{B}.$ (10)

The transformation that aligns the axes can be computed using the Kabsch
algorithm [45] which minimizes a root-mean-square error function to find the
optimal rotation matrix that aligns a set of vectors. The keen observer will
note that this alignment does not quite yield the configuration desired. In
fact, it aligns the coordinate systems such that the human would be facing
away from the robot’s camera. To anti-align the coordinate systems, we define
$\tilde{\text{T}}\leftarrow\text{R}_{y}(\pi)\tilde{\text{T}}$. This produces
the desired rotation.

Together, along with a translation constraint, which defines how close the
keypoints should appear to the camera frame, or the robot, these can be used
to compute the components of the transformation matrix. Finally, perspective
projection using the camera intrinsics yields the appearance of the points in
the image plane by $\vec{s}^{*}=\text{K}\tilde{\text{T}}\vec{k}$. The vector
of points $\vec{s}^{*}$ is the scale-preserved setpoint.

The benefits of this approach are two-fold. By computing a body-fixed frame,
we can compute the setpoint for a visual-servo control scheme that is scale-
aware. This means that different diver body shapes will appear as different
sizes depending on the scale heuristic. For example, the divers shown in Fig.
6 are at the same distance from the camera, but the ideal setpoint is very
different.

(a)

(b)

Figure 6: Two divers of different body shapes have different setpoints at the
same scale, in this case 2 m from the camera. The crosses in the right-hand
figure are the baseline setpoints from the diver in the left-hand image. The
points have been shifted to center on the mean of the keypoints in the right-
hand image. F2F produces scale aware setpoints, such that the robot can
utilize visual servo techniques to ensure safe approach from non-standard body
poses.

As a result, the control scheme would indicate that the robot should move
closer to achieve the same level of error between the setpoint and the
observed pose. Of course, this could potentially place the diver in harms way
if the robot malfunctions.

##  4 Experimental Results

Experiments were conducted using data collected from divers in both closed-
water (_i.e._ , pool) settings and in the ocean waters off the coast of
Barbados, West Indies. For this work, we focus on measuring the error between
baseline setpoints and constructed setpoints using our alignment vector
convention. We also provide additional analysis of different pose estimation
networks on our custom dataset in the accompanying video submission.

(a) 1 m

(b) 2 m

(c) 3 m

Figure 7: Setpoints used as baselines for comparing projections from reconstruction and alignment. Note that setpoint baselines (colored dots) have been shifted to the center of the image to accommodate for differences in image capture that occurred due to strong ocean currents that made station keeping challenging for the divers. Table 1: Summary of projection errors between setpoint baselines at $1$, $2$, and $3$ m distances and projections from scale-preserving computations, averaged over $50$ projection estimates. Errors are reported as mean$\pm$standard deviation in pixel units. Each error is the sum over all keypoints of the Euclidean distance between the ground truth and the predicted keypoint location. Pose |  $\boldsymbol{1}~{}\textbf{m}$ |  $\boldsymbol{2}~{}\textbf{m}$ |  $\boldsymbol{3}~{}\textbf{m}$ |  Across distances
---|---|---|---|---
Prone (surface) |  $175.39\pm 62.67$ |  $227.19\pm 8.3$ |  $322.61\pm 0.0$ |  $241.73\pm 23.66$
Prone (bottom) |  $181.43\pm 64.31$ |  $192.04\pm 0.0$ |  $157.98\pm 6.51$ |  $177.15\pm 23.6$
Upright (away) |  $367.4\pm 125.43$ |  $61.9\pm 0.98$ |  $88.81\pm 0.0$ |  $173.99\pm 52.7$
Upright (facing) |  $166.84\pm 0.0$ |  $383.12\pm 157.83$ |  $92.67\pm 31.70$ |  $214.21\pm 63.17$
Inverted (facing) |  $176.71\pm 43.63$ |  $101.43\pm 12.77$ |  $224.08\pm 71.63$ |  $167.41\pm 42.68$
Inverted (away) |  $366.11\pm 36.35$ |  $172.28\pm 43.43$ |  $131.31\pm 58.75$ |  $223.23\pm 46.18$
Across poses |  $238.98\pm 55.40$ |  $189.66\pm 37.22$ |  $170.22\pm 33.38$ |

(a) Prone (surface)

(b) Prone (bottom)

(c) Upright (facing)

(d) Upright (away)

(e) Inverted (facing)

(f) Inverted (away)

Figure 8: The projections we achieve through the alignment system are
reasonable, given the challenges of reconstruction. The cross markers indicate
the $1$ m setpoint baseline computed from sea trial data. The large dots
indicate projections from the alignment system. A standard deviation of $0.0$
indicates instances in which the system was unable to triangulate accept for a
single instance during the frame acquisition.

To collect setpoint baselines, a diver for whom we had existing pose data was
asked to station keep above an experimental trackline. The trackline was
measured to $1$, $2$, and $3$ m distances. The camera operator asked the diver
to spend approximately $15$ seconds station keeping during acquisition of
stereo image data. The camera operator then signaled the diver to move
forward. This process was repeated until the diver had been recorded at all
three distances. The setpoint baseline was computed by using a simple by-hand
label technique that mimicked what a field operator or end-user would do
during calibration of a visual servo system. This hand measurement is shown at
the three distances in Fig. 7.

We then performed experiments using the setpoints extracted from the three
distances and comparing the euclidean error and standard deviation of the
error between six canonical pose positions. These results are summarized in
the table below. Note that $50$ frames of data at each canonical pose were
used during the error computation. A reasonable expectation might be on the
order of the error of the pose estimation network. In this case, approximately
$60$ pixels of error across all projected keypoints is reasonable. Clearly we
do not observe that low of error, except in exceptional cases. There are a
couple of reasons for this. The pose estimation network runs inference on both
the left- and right-hand image pairs, and any fluctuations caused from camera
motion or inconsistent lighting conditions can cause significant errors during
triangulation. To that end, notice that we have also shown not only the
average error in the figures of Fig. 8 but also the minimum error observed
during the frame acquisition. Some poses appear exceptionally good
qualitatively and tend to agree with the baselines. There is a trend that
indicates reconstruction and projection is better at distances that exceed $1$
m. This is likely caused by better pose estimations. Most of the dataset on
which we trained DLC was collected at distances of approximately $2$–$3$ m
from the diver. As a result, predictions from the pose network are better and
more consistent at these distances, likely causing reductions in
reconstruction errors throughout the frames used for analysis.

The results summarized in Table 1 demonstrate that we achieve reasonable
projection errors for most poses. The visual results for the $1$ m poses are
shown in Figure 8 for the six canonical pose states.

##  5 Future Work

Scale-preservation and automatic setpoint computation allows a robot to
autonomously detect the desired keypoint configuration, without need for prior
calibration. This means in a complex mission, the robot has the ability to re-
orient itself with respect to the diver to establish F2F communication.
However, the work described has shortcomings. First, totally occluded joints
pose a problem for reconstruction, because lateral views appear at the same
distance. To subvert this, anthropometric data ratios (ADRs), which describe
ratios between human limbs, can be used to regularize depth estimates so that
totally lateral views do not appear to have joints at the same distance.
Another issue with the present approach is that the algorithm does not account
for instances where the diver’s body is positioned behind an obstacle relative
to the camera’s origin. Future work will include context-aware approach
strategies that utilize the entirety of the scene for understanding the
diver’s behavior and selecting the best place to approach from, particularly
with regard to complex mission tasks. We will conduct control experiments
using the F2F system in pool environments, where evaluation of the control can
be better constrained. The ocean environment is susceptible to external
perturbations from currents making it challenging to evaluate the efficacy of
the F2F system.

##  6 Conclusion

We have demonstrated that our methodology enables automatic setpoint
computation from human diver pose observations, which could permit more
complex UHRI scenarios where the robot must infer automatically the best
orientation of target features to navigate within a safe distance of the human
diver. We have shown that our method works well for $1$, $2$, and $3$ m
baselines, except in cases of poses in which a joint is totally occluded and
the triangulation is unable to resolve the depth at which the occluded joint
appears.

## References

  * [1] M. Matarić, “On relevance: Balancing theory and practice in HRI,” ACM Transactions on Human-Robot Interaction (THRI), vol. 7, no. 1, p. 1–2, 2018.
  * [2] A. Cherubini, R. Passama, A. Crosnier, A. Lasnier, and P. Fraisse, “Collaborative manufacturing with physical human–robot interaction,” Robotics and Computer-Integrated Manufacturing, vol. 40, p. 1–13, 2016.
  * [3] H. Su, J. Sandoval, M. Makhdoomi, G. Ferrigno, and E. De Momi, “Safety-enhanced human-robot interaction control of redundant robot for teleoperated minimally invasive surgery,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), p. 6611–6616, IEEE, 2018.
  * [4] J. Fasola and M. J. Matarić, “A socially assistive robot exercise coach for the elderly,” Journal of Human-Robot Interaction, vol. 2, no. 2, p. 3–32, 2013.
  * [5] F. Jentsch, Human-robot interactions in future military operations.  CRC Press, 2016.
  * [6] M. Fulton, C. Edge, and J. Sattar, “Robot communication via motion: Closing the underwater human-robot interaction loop,” in 2019 International Conference on Robotics and Automation (ICRA), p. 4660–4666, IEEE, 2019.
  * [7] L. Takayama, W. Ju, and C. Nass, “Beyond dirty, dangerous and dull: what everyday people think robots should do,” in 2008 3rd ACM/IEEE international conference on human-robot interaction (HRI), p. 25–32, IEEE, 2008.
  * [8] T. B. Sheridan, “Human–robot interaction: status and challenges,” Human factors, vol. 58, no. 4, p. 525–532, 2016.
  * [9] P. Corke, C. Detweiler, M. Dunbabin, M. Hamilton, D. Rus, and I. Vasilescu, “Experiments with underwater robot localization and tracking,” in Proceedings 2007 IEEE International Conference on Robotics and Automation, p. 4556–4561, IEEE, 2007.
  * [10] C. Edge, M. J. Islam, C. Morse, and J. Sattar, “A generative approach for detection-driven underwater image enhancement,” arXiv preprint arXiv:2012.05990, 2020.
  * [11] M. J. Islam, Y. Xia, and J. Sattar, “Fast underwater image enhancement for improved visual perception,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3227–3234, 2020.
  * [12] M. J. Islam, P. Luo, and J. Sattar, “Simultaneous Enhancement and Super-Resolution of Underwater Imagery for Improved Visual Perception,” in Proceedings of Robotics: Science and Systems, (Corvalis, Oregon, USA), July 2020.
  * [13] M. C. Hastings, “Coming to terms with the effects of ocean noise on marine animals,” Acoustics today, vol. 4, no. 2, p. 22–34, 2008.
  * [14] M. Fulton, A. Prabhu, and J. Sattar, “HREyes: Design, Development, and Evaluation of a Novel Method for AUVs to Communicate Information and Gaze Direction,” arXiv preprint arXiv:2211.02946, 2022.
  * [15] S. S. Enan, M. Fulton, and J. Sattar, “Robotic Detection of a Human-Comprehensible Gestural Language for Underwater Multi-Human-Robot Collaboration,” arXiv preprint arXiv:2207.05331, 2022.
  * [16] M. Fulton, J. Hong, and J. Sattar, “Using Monocular Vision and Human Body Priors for AUVs to Autonomously Approach Divers,” in 2022 International Conference on Robotics and Automation (ICRA), p. 1076–1082, IEEE, 2022.
  * [17] S. S. Enan and J. Sattar, “Visual Detection of Diver Attentiveness for Underwater Human-Robot Interaction,” arXiv preprint arXiv:2209.14447, 2022.
  * [18] Y. Xia and J. Sattar, “Visual diver recognition for underwater human-robot collaboration,” in 2019 international conference on robotics and automation (ICRA), p. 6839–6845, IEEE, 2019.
  * [19] M. J. Islam and J. Sattar, “Mixed-domain biological motion tracking for underwater human-robot interaction,” in 2017 IEEE international conference on robotics and automation (ICRA), p. 4457–4464, IEEE, 2017.
  * [20] J. Sattar and G. Dudek, “Where is your dive buddy: tracking humans underwater using spatio-temporal features,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 3654–3659, IEEE, 2007.
  * [21] I. Kvasić, N. Mišković, and Z. Vukić, “Convolutional neural network architectures for sonar-based diver detection and tracking,” in OCEANS 2019-Marseille, p. 1–6, IEEE, 2019.
  * [22] K. W. Lo and B. G. Ferguson, “Diver detection and localization using passive sonar,” in Proceedings of Acoustics, vol. 8, p. 1–8, 2012.
  * [23] K. J. DeMarco, M. E. West, and A. M. Howard, “Sonar-based detection and tracking of a diver for underwater human-robot interaction scenarios,” in 2013 IEEE International Conference on Systems, Man, and Cybernetics, p. 2378–2383, IEEE, 2013.
  * [24] M. Fulton, C. Edge, and J. Sattar, “Robot Communication Via Motion: A Study on Modalities for Robot-to-Human Communication in the Field,” ACM Transactions on Human-Robot Interaction (THRI), vol. 11, no. 2, p. 1–40, 2022.
  * [25] G. Dudek, J. Sattar, and A. Xu, “A Visual Language for Robot Control and Programming: A Human-Interface Study,” in Proceedings of the International Conference on Robotics and Automation ICRA, (Rome, Italy), pp. 2507–2513, April 2007.
  * [26] A. Xu, G. Dudek, and J. Sattar, “A Natural Gesture Interface for Operating Robotic Systems,” in Proceedings of the IEEE International Conference on Robotics and Automation, ICRA, (Pasadena, California), pp. 3557–3563, May 2008.
  * [27] A. Sagitov, K. Shabalina, L. Sabirova, H. Li, and E. Magid, “ARTag, AprilTag and CALTag Fiducial Marker Systems: Comparison in a Presence of Partial Marker Occlusion and Rotation.,” in ICINCO (2), p. 182–191, 2017.
  * [28] H. Buelow and A. Birk, “Gesture-recognition as basis for a human robot interface (hri) on a auv,” in OCEANS’11 MTS/IEEE KONA, pp. 1–9, Sept 2011.
  * [29] D. Chiarella, M. Bibuli, G. Bruzzone, M. Caccia, A. Ranieri, E. Zereik, L. Marconi, and P. Cutugno, “A novel gesture-based language for underwater human–robot interaction,” Journal of Marine Science and Engineering, vol. 6, no. 3, p. 91, 2018.
  * [30] M. J. Islam, M. Ho, and J. Sattar, “Understanding human motion and gestures for underwater human–robot collaboration,” Journal of Field Robotics, vol. 36, no. 5, p. 851–873, 2019.
  * [31] A. Gomez Chavez, A. Ranieri, D. Chiarella, and A. Birk, “Underwater Vision-Based Gesture Recognition: A Robustness Validation for Safe Human–Robot Interaction,” IEEE Robotics and Automation Magazine, vol. 28, no. 3, pp. 67–78, 2021.
  * [32] B. Verzijlenberg and M. Jenkin, “Swimming with Robots: Human Robot Communication at Depth,” in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4023–4028, 2010.
  * [33] M. Bernardi, C. Cardia, P. Gjanci, A. Monterubbiano, C. Petrioli, L. Picari, and D. Spaccini, “The Diver System: Multimedia Communication and Localization Using Underwater Acoustic Networks,” in 2019 IEEE 20th International Symposium on “A World of Wireless, Mobile and Multimedia Networks” (WoWMoM), pp. 1–8, 2019.
  * [34] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning.  MIT press, 2016.
  * [35] B. Xiao, H. Wu, and Y. Wei, “Simple baselines for human pose estimation and tracking,” in Eur. Conf. Comput. Vis., p. 466–481, 2018.
  * [36] H. Dai, H. Shi, W. Liu, L. Wang, Y. Liu, and T. Mei, “FasterPose: A Faster Simple Baseline for Human Pose Estimation,” ACM TOMM, vol. 18, no. 4, p. 1–16, 2022.
  * [37] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime multi-person 2D pose estimation using part affinity fields,” in IEEE Conf. Comput. Vis. Pattern Recog., p. 7291–7299, 2017.
  * [38] Z. Yu, J. S. Yoon, I. K. Lee, P. Venkatesh, J. Park, J. Yu, and H. S. Park, “Humbi: A large multiview dataset of human body expressions,” in IEEE Conf. Comput. Vis. Pattern Recog., p. 2990–3000, 2020.
  * [39] J. S. Yoon, Z. Yu, J. Park, and H. S. Park, “HUMBI: A Large Multiview Dataset of Human Body Expressions and Benchmark Challenge,” arXiv preprint arXiv:2110.00119, 2021.
  * [40] F. Chaumette and S. Hutchinson, “Visual servo control. I. Basic approaches,” IEEE Robotics & Automation Magazine, vol. 13, no. 4, p. 82–90, 2006.
  * [41] F. Chaumette and S. Hutchinson, “Visual servoing and visual tracking,” 2008.
  * [42] F. Chaumette, S. Hutchinson, and P. Corke, “Visual servoing,” in Springer Handbook of Robotics, p. 841–866, Springer, 2016.
  * [43] A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis, and M. Bethge, “DeepLabCut: markerless pose estimation of user-defined body parts with deep learning,” Nature neuroscience, vol. 21, no. 9, p. 1281–1289, 2018.
  * [44] T. Nath, A. Mathis, A. C. Chen, A. Patel, M. Bethge, and M. W. Mathis, “Using DeepLabCut for 3D markerless pose estimation across species and behaviors,” Nature protocols, vol. 14, no. 7, p. 2152–2176, 2019.
  * [45] W. Kabsch, “A Discussion of the Solution for the Best Rotation to Relate Two Sets of Vectors,” Acta Crystallographica Section A, vol. 34, pp. 827–828, Sep 1978.