# Fast One-Stage Unsupervised Domain Adaptive Person Search

Tianxiang Cui1 Huibing Wang1 Corresponding author Jinjia Peng2 Ruoxi Deng3
Xianping Fu1 Yang Wang4∗
1The College of Information and Science Technology, Dalian Maritime
University, China
2The College School of Cyber Security and Computer, Hebei University, China
3The College of Computer Science and Artificial Intelligence, Wenzhou
University, China
4Key Laboratory of Knowledge Engineering with Big Data, Hefei University of
Technology, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Unsupervised person search aims to localize a particular target person from a
gallery set of scene images without annotations, which is extremely
challenging due to the unexpected variations of the unlabeled domains.
However, most existing methods dedicate to developing multi-stage models to
adapt domain variations while using clustering for iterative model training,
which inevitably increases model complexity. To address this issue, we propose
a Fast One-stage Unsupervised person Search (FOUS) which complementary
integrates domain adaptaion with label adaptaion within an end-to-end manner
without iterative clustering. To minimize the domain discrepancy, FOUS
introduced an Attention-based Domain Alignment Module (ADAM) which can not
only align various domains for both detection and ReID tasks but also
construct an attention mechanism to reduce the adverse impacts of low-quality
candidates resulting from unsupervised detection. Moreover, to avoid the
redundant iterative clustering mode, FOUS adopts a prototype-guided labeling
method which minimizes redundant correlation computations for partial samples
and assigns noisy coarse label groups efficiently. The coarse label groups
will be continuously refined via label-flexible training network with an
adaptive selection strategy. With the adapted domains and labels, FOUS can
achieve the state-of-the-art (SOTA) performance on two benchmark datasets,
CUHK-SYSU and PRW. The code is available at https://github.com/whbdmu/FOUS

##  1 Introduction

The objective of the pedestrian search task is to identify and locate specific
individuals within panoramic images. Initially, all pedestrians present in the
image are detected, and subsequently, they are compared to the target
pedestrian image for recognition. In the context of the intelligent security
field, it becomes increasingly difficult and costly to complete person search
tasks in a supervised manner due to the massive volume of videos and pictures
generated by numerous cameras. Therefore, unsupervised domain adaptive methods
should receive more attention.

(a) Clustering-based person search

(b) Ours

Figure 1: Differences in operation between our proposed method and mainstream
methods. 1(a) represents that the clustering algorithm utilizes the original
data to calculate similarity in each iteration. 1(b) illustrates that our
proposed method utilizes the original data only in the first iteration to
assign soft labels, and then gradually refines the labels.

The main challenges currently in unsupervised pedestrian search are the lack
of bounding box annotations and the lack of pedestrian identity label
annotations. Since it is very difficult to directly train unsupervised
detection and re-identification, a series of weakly supervised research
worksYan et al. (2022); Han et al. (2021); Jia et al. (2023) have been
proposed to address the challenge of lacking identity annotations, which have
employed clustering techniques to assign pseudo labels. As shown in Fig.1(a),
clustering approaches typically require multiple iterations of calculating
sample similarity, which can be time-consuming and resource-intensive.
Moreover, these methods still demand significant time and effort to annotate
ground-truth bounding boxes, and their model generalization capability is
limited, rendering them unsuitable for real-world scenarios.

Figure 2: The design architecture of the FOUS framework. For each iteration,
after extracting features, FOUS alternates between two phases: (1) Attention-
based Domain Alignment.(Sec. 2.2) The candidate box quality is enhanced
through the Multi-Information Perception Attention Module, followed by domain
alignment operations. (2) Unlabeled Target Domain Training.(Sec. 2.3) Annotate
the unlabeled samples in the target domain utilizing prototypes as the
reference for further fine-tuning, in which select random features as the
target prototype vectors and update the learned source prototypes.

Therefore, unsupervised person search was emerged as the times require.
Considering the difficulty and impracticality of directly training
unsupervised detection and re-identification, current research methods rely on
powerful pre-trained models. Li et al.[4] introduced a domain adaptive method
to address the challenge of unsupervised pedestrian search by training on a
labeled dataset from the source domain. Specifically, Li et al. designed a
domain alignment module to alleviate the difference between unsupervised
detection and re-identification subtasks, and then proposed a dynamic
clustering strategy to generate pseudo labels on the target domain. We can
observe that the clustering method, which is widely employed in both weakly
supervised and unsupervised person re-identification tasks, offers certain
advantages. It is capable of withstanding noisy labels and generating
dependable labels, thus effectively training a model with robust
generalization capabilities. However, the computational resources required for
this method are quite demanding. The clustering method involves calculating
pairwise similarities and multiple iterations to optimize results, which can
result in an impractical computational burden. Consequently, these methods
face challenges when applied to smart security scenarios involving a massive
amount of video footage.

In order to solve this problem, this paper proposes a fast and efficient
unsupervised domain adaptation framework for the person search task, which can
learn a model with strong generalization ability at a lower cost. As
illustrated in Fig.1(b), different from the clustering methods of existing
models, FOUS gives up generating relatively accurate labels through complex
clustering algorithms, and instead tries to learn potential correlations from
soft labels with greater noise and fewer resources. First, in order to quickly
assign soft labels, we use the prototype correlation labeling method to
calculate the similarity between partial samples only once. To reduce
computational resources, these prototypes are used as annotators to soft-label
unlabeled samples based on nearest neighbor correlations. Secondly,
considering the importance of mitigating the impact of generated noisy labels,
we follow the domain alignment module in DAPS and introduce a new attention
mechanism, which can amplify global-dimensional interaction features while
reducing information dispersion. Effectively reduce the number of noisy labels
generated by low-quality candidate frames in unsupervised detection. And FOUS
also proposes an adaptive selection strategy to perform label-flexible
training of the network to gradually refine the rough labels.

Our contributions can be summarized as follows:

1\. We introduce a fast unsupervised domain-adaptive pedestrian search
framework based on a prototype correlation labeling method instead of the
prevalent clustering algorithm. Compared with multiple iterations of the
clustering algorithm, FOUS only calculates the similarity between some samples
once and directly assigns soft label groups.

2\. In order to reduce the noise in the soft label group, we first introduce a
new attention mechanism to alleviate the differences between unsupervised
detection and re-identification tasks, and also propose an adaptive selection
strategy to gradually refine the coarse labels to find reliable and similar
target images.

3\. Without any auxiliary labels in the target domain, our method achieves
good performance on both pedestrian search benchmark datasets, which not only
reduces the amount of calculation and improves the inference speed but also
has good generalization ability.

##  2 Method

###  2.1 Framework Overview

Fig.2 highlights the design architecture of the proposed FOUS framework. Given
the input images from both the source and the target domain, we utilize a pre-
trained model on the source domain annotation dataset to produce the source-
prototype vector, and the image-level feature maps are extracted with a
backbone network. Next, these features are fed into the Region Proposal
Network (RPN) to generate proposals, which are passed through the attention
module to create instance-level feature maps. To bridge the domain gaps and
ensure effective detection and ReID (Re-Identification) tasks, we introduce an
attention-based domain alignment module (ADAM). The ADAM is designed to align
both image-level and instance-level features across different domains. It
helps harmonize the representations and close the gap between domains.

To tackle the issue of computational cost in unsupervised reID tasks, a new
prototype-guided labeling method is proposed to assign and optimize soft
labels instead of computationally complex clustering methods, which frees up
the process of label generation, including prototypes-guided labeling phase
and label-flexible training phase. We first annotate the unlabeled samples
with coarse labels in a simple way, which selects the prototype vectors
generated above and softly annotates unlabeled samples via the relevance
between these prototypes and samples. Additionally, FOUS trains its models
using a combination of cluster-level contrastive loss and instance-level loss,
which ensures that the models adhere to both the constraints imposed by the
two types of losses. It aims to refine the coarse labels by incorporating the
proposed adaptive selection strategy. Above all, except for the initial
iteration, the labeled samples from the source domain are not utilized in our
FOUS.

Figure 3: The channel-level information aggregation and spatial-level
information aggregation in ADAM.

###  2.2 Attention-based Domain Alignment Module

Attention Mechanism. To mitigate the negative effects of low-quality proposals
that arise from unsupervised detection, a novel attention module is designed,
which can amplify global-dimensional interactive features while reducing
information dispersion. Fig.2 emphasizes that the attention module primarily
consists of information aggregation and information interaction.

Input feature $x\in\mathbb{R}^{H\times W\times C}$ is initially processed into
feature $x^{c}\in\mathbb{R}^{H\times W\times C}$ and feature
$x^{s}\in\mathbb{R}^{H\times W\times C}$ through channel-level information
aggregation and spatial-level information aggregation respectively. And these
two features are subsequently cross-connected in the channel dimension to
yield features $x^{cs}\in\mathbb{R}^{H\times W\times 2C}$. The information
aggregation is formulated as:

$x^{cs}=Concat(f_{clia}(x),f_{slia}(x)),$ (1)

where $f_{clia}$, $f_{slia}$ and $Concat$ represent channel-level information
aggregation, spatial-level information aggregation and cross-concatenation,
respectively. Channel-level information aggregation can capture channel
dependency information, channel structure information, and spatial global
information through cross-channel global covariance pooling and channel-level
global average pooling. Spatial-level information aggregation can capture
spatial dependency information, spatial structure information, and channel
global information through cross-spatial global covariance pooling and
spatial-level global average pooling. Cross-concatenation plays an important
role in organizing various pieces of information, enabling effective
interaction and integration of subsequent information.

Fig.3 illustrates the specific structure of channel-level information
aggregation and spatial-level information aggregation in ADAM, which can
simultaneously achieve the perception of multi-dimensional dependency
information, multi-dimensional global information and multi-dimensional
structural information in information aggregation. The channel-level
information aggregation is formulated as follows:

${f_{clia}}(x)=Up\left({MM\left({CcGCP(x),CwGAP(x)}\right)}\right)$ (2)

where $CcGCP$, $CwGAP$, $MM$ and $Up$ represent cross-channel global
covariance pooling, channel-wise global average pooling, matrix multiplication
and upsampling operations, respectively. Moreover, the spatial-level
information aggregation is formulated as follows:

${f_{slia}}(x)=Up\left({MM\left({SwGAP{{(x)}^{\Delta}},CsGCP{{(x)}^{\Delta}}}\right)}\right)$
(3)

where $SwGAP$, $CsGCP$ and $\Delta$ represent spatial-wise global average
pooling, cross-spatial global covariance pooling and reshape operations,
respectively.

Similarly, feature $x^{cs}\in\mathbb{R}^{H\times W\times 2C}$ is separated
into feature $x^{c^{\prime}s}\in\mathbb{R}^{H\times W\times C}$ and feature
$x^{cs^{\prime}}\in\mathbb{R}^{H\times W\times C}$ through channel-driven
information interaction and spatial-driven information interaction
respectively. And these two features are subsequently adaptively fused into
features $x^{c^{\prime}s^{\prime}}\in\mathbb{R}^{H\times W\times C}$ by
allocating learnable parameters $\alpha\in\mathbb{R}^{1\times 1\times
C}\mathbb{}$ and $\beta\in\mathbb{R}^{1\times 1\times C}$. The information
interaction is formulated as:

$x^{c^{\prime}s^{\prime}}=\alpha f_{cdii}(x^{cs})+\beta f_{sdii}(x^{cs}),$ (4)

where $f_{cdii}$ and $f_{sdii}$ represent channel-driven information
interaction and space-driven information interaction, respectively. Channel-
driven interaction focuses on perceiving diversity in channels by assigning
unique parameters along the channel dimension while sharing the same
parameters in the spatial dimension. On the other hand, spatial-driven
interaction aims to perceive diversity in space by assigning distinct
parameters along the spatial dimension while sharing the same parameters in
the channel dimension.

Fig.4 shows the specific structure of channel-level information interaction
and spatial-level information interaction in ADAM, which concurrently achieves
the perception of multidimensional diverse information during information
interaction. ADAM stimulates intrinsic information potential through a more
comprehensive proactive interaction. The channel-level information interaction
is formulated as follows:

${f_{cdii}}({x^{cs}})=GCon{v_{++}}({x^{cs}})$ (5)

where $GConv$ represents the combination of group convolution, batch
normalization, and the ReLU function. The information interaction between
different groups is undisturbed. Channel-driven information interaction,
achieved by assigning different parameters in the channel dimension and
sharing the same information parameters in the spatial dimension, not only
facilitates the interaction and fusion of multiple information in channel-
level and spatial-level information collection but also further perceives
channel diversity information.

The spatial-level information interaction is formulated as follows:

${f_{sdii}}({x^{cs}})=Concat\left({Sum{{\left({GCon{v_{++}}(Split({x^{cs\Delta}}))}\right)}^{\Delta}}}\right)$
(6)

where $Sum$ represents the row-wise summation. Reshaping and splitting
operations are used to adjust the feature shape for subsequent specific
information interactions. The spatial-driven information interaction, achieved
by assigning different parameters in the spatial dimension and sharing the
same parameters in the channel dimension, not only facilitates the interaction
and fusion of multiple information from channel-level and spatial-level
information collection but also further perceives spatial diversity
information.

Finally, feature $x^{c^{\prime}s^{\prime}}\in\mathbb{R}^{H\times W\times C}$
is activated into the attention map by the sigmoid function. The attention map
is employed to modulate input sub-feature $x\in\mathbb{R}^{H\times W\times
C}$, resulting in the output sub-feature $x^{\prime}\in\mathbb{R}^{H\times
W\times C}$:

$x^{\prime}=x\mathcal{S}{igmoid}(x^{c^{\prime}s^{\prime}})$ (7)

The sigmoid function distinguishes the importance by activating eigenvalues
within the range of 0 to 1. Importantly, the input sub-features are
transformed into output sub-features across all branches. This multi-branch
structure is advantageous in activating different attention mechanisms,
allowing for the selective perception of valuable feature information on each
branch in a targeted manner.

Figure 4: The channel-level information interaction and spatial-level
information interaction in ADAM.

Task-sensitive Domain Alignment. As studied by Chen et al. (2018b); You et al.
(2019); Xu et al. (2020); Wang et al. (2024), reducing domain disparities is
advantageous for both subtasks within person search, and can enhance the
model’s capacity to learn domain-invariant representations. Given the swift
advancement of domain-adaptive detectorsKhodabandeh et al. (2019); Wang
(2021); Vibashan et al. (2023); Zhang et al. (2023), FOUS proposes a task-
sensitive domain alignment module. As shown in Fig.2, ADAM uses a patch-based
domain classifier to predict the domain from which the input features come,
and uses a minmax formula to mislead the domain classifier and encourage
domain-invariant representation learning. Assuming we possess $N$ training
images $\left\\{I_{1},...,I_{N}\right\\}$ along with their respective domain
labels, where $0$ and $1$ represent the source domain and target domain
respectively. Then denote the domain classifier as $D$, and further denote the
domain prediction result of the input feature $I_{i}$ as $p_{i}$. We
incorporate a cross-entropy loss for domain alignment through an adversarial
training approach:

$L_{dom}=-\sum_{i}[d_{i}logp_{i}+(1-d_{i})log(1-p_{i})]$ (8)

Based on the characteristics of the upstream and downstream tasks, we suggest
achieving balance in the task-sensitive alignment module by equitably
adjusting the alignment weights of instance-level features in both sub-tasks.
The instance-level loss can be formulated as:

$\displaystyle L_{ins}=$
$\displaystyle-\lambda\sum_{i,j}[d_{i}logp_{i,j}^{d}+(1-d_{i})log(1-p_{i,j}^{d})]$
(9)
$\displaystyle-(1-\lambda)\sum_{i,k}[d_{i}logp_{i,k}^{r}+(1-d_{i})log(1-p_{i,k}^{r})]$

where $j\in\left\\{1,...,K_{1}\right\\}$ and
$k\in\left\\{1,...,K_{2}\right\\}$. The source domain comprises $N_{s}$
images, while the target domain includes $N_{t}$ images. The balancing factor
$\lambda$ is set as:

$\lambda=\sigma\left(4\cdot
sign(N_{t}-N_{s})\left(\frac{\max(N_{s},N_{t})}{\min(N_{s},N_{t})}-1\right)\right)$
(10)

where $\sigma$ is Sigmoid function employed for domain normalization. What’s
more, we introduce an L2-norm regularizer to maintain consistency between
image-level and instance-level classifiers.

###  2.3 Unlabeled Target Domain Training

FOUS employs a prototype-relevance labeling method as a replacement for
complex clustering algorithms, leading to a significant reduction in
computational costs. In FOUS, the prototype vector needs to be initially
selected or calculated, followed by the computation of the correlation between
the prototype vector and all the samples. Then assign the label of the nearest
prototype vector to the sample as a pseudo label. Therefore, the selection of
the prototype vector is a critical aspect. And FOUS design two types of
prototypes, namely the source-prototype vector and the random-prototype
vector. Initially, the source-prototype vector is computed using well-labeled
data from the source domain. Subsequently, to enhance FOUS’s adaptability in
the target domain, features are randomly selected from the target domain to
create a random-prototype vector.

Prototype vector initialization We utilize the network $E_{s}(\cdot)$, pre-
trained on the source domain, to extract features from the source domain
samples. Subsequently, based on the ground-truth labels, the source-prototype
vector can be computed as:

$P_{k}^{s}=\frac{1}{N_{k}}\sum_{f_{i}\in F_{k}^{s}}f_{i}$ (11)

where $F_{k}^{s}$ represents a set of features with the same labels $k$.
$f_{i}$ is the feature of each instance. $N_{k}$ is the number of samples with
the label $k$. The source-prototype vector $P^{s}$ is composed of
$\left\\{P_{1}^{s},...,P_{K}^{s}\right\\}$, $k\in[1,K]$. Particularly, $K$
means the number of identity categories in the source domain. However, models
trained solely on pseudo-labels generated from source-prototype vectors tend
to be sensitive to variations in the target domain and are susceptible to
introducing a considerable amount of noise. Hence, FOUS sets up random
prototype vectors in the target domain to enhance diversity. Similar to the
source-prototype, $E_{s}(\cdot)$ is employed to extract features from target
domain samples. Subsequently, $N_{t}$ vectors are randomly selected to form
the random-prototype $P^{t}=\left\\{P_{1}^{t},...,P_{N_{t}}^{t}\right\\}$.

Labeling Once the source and random prototype vector are calculated, the
labels for other sample instances in the target domain are determined by the
k-Nearest Neighbor algorithm,

$l_{i}^{s}=\mathop{\arg\min}\limits_{l^{s}}dist(f_{i},P^{s})$ (12)

where $dist(a,b)$ means the Euclidean distance between $a$ and $b$. $f_{i}$ is
the characteristic of the $i$-th instance. Eq.12 signifies assigning the
instance with the label of the vector in $P^{s}$ that is closest to $f_{i}$.
All the labeled images collectively constitute a new dataset $D^{s}$. Similar
to $D^{s}$, according to the random-prototype $P^{t}$, we can also derive a
group of labels,

$l_{i}^{t}=\mathop{\arg\min}\limits_{l^{t}}dist(f_{i},P^{t})$ (13)

The assignment of pseudo labels through prototypes results in the creation of
two datasets, which are
$D^{s}=\left\\{(x_{i},l_{1}^{s},...,l_{n}^{s})\right\\}_{i=1}^{N}$ and
$D^{t}=\left\\{(x_{i},l_{1}^{t},...,l_{n}^{t})\right\\}_{i=1}^{N}$, $n$
represents the number of instances in $x_{i}$. Then, the source-prototype is
updated by the target datasets with pseudo labels,

$P_{j}^{s}=\frac{1}{N_{j}}\sum_{f\in F_{j}^{t}}f$ (14)

where $F_{j}^{t}$ means the features belonging to the cluster $j$. Therefore,
FOUS does not use images from the source domain except for the first
iteration.

Label-flexible Training To improve the training of models with flexible
labels, FOUS constructs distinct memory structures to store features based on
different label groups. Based on the source prototype $P^{s}$ and random
prototype $P^{t}$, the pseudo labels $L^{s}$ and $L^{t}$ can be calculated. To
mitigate the influence of noise, we designed two cluster-level memory
structures to store the features at the training stage. Taking the $L^{s}$ as
an example, the features $\left\\{c_{1},...c_{K}\right\\}$ of each cluster are
stored in a memory-based feature dictionary. $K$ means the number of clusters.
After clustering with prototypes, the features in the cluster are utilized to
initialize the memory dictionary, which could be expressed as,

$m_{k}=\frac{1}{N_{k}}\sum_{f\in F_{k}}f$ (15)

where $F_{k}$ represents the feature set of instances labeled as $k$,
comprising all feature vectors in cluster $k$. $N_{k}$ is the number of
features in $F_{k}$. When employing random-guided memory, we are provided with
a query instance $q$ and perform feature comparisons with all features within
the respective memory$\left\\{m_{1}^{t},...m_{K}^{t}\right\\}$ structure
utilizing the InfoNCE loss,

$L_{c}^{t}=-\log\frac{\exp(sim(f\cdot
m_{+}^{t})/\tau)}{{\textstyle\sum_{k=1}^{K}}\exp(sim(f\cdot m_{k}^{t})/\tau)}$
(16)

where $m_{+}^{t}$ denotes the features that share the same identity as f
within the random-guided memory. $\tau$ is the proportional coefficient
balancing factor. Similar to Eq.16, the loss $L_{c}^{s}$ of the source-guided
memory also could be calculated.

In addition, FOUS employs invariant learning between images to bolster the
model’s generalization capability. An instance-level memory bank
$\left\\{m_{1}^{l},...m_{N}^{l}\right\\}$ is consequently established to store
all features, and soft labels are estimated by evaluating the correlation
between images.

Due to the absence of genuine labels, neighborhoods displaying high
correlation play a role in diminishing the distance between similar images.
Consequently, FOUS introduces an adaptive selection strategy. Taking
inspiration from Ding et al. (2020); Peng et al. (2020); Wang et al. (2022), a
similarity threshold is defined. Only when the distance between the adaptively
chosen neighbors and the provided image falls below this threshold, these
samples are presumed to share the same label as the target image. Therefore,
these samples should exhibit closer proximity to the target image. The
corresponding loss can be formulated as:

$L_{t}^{e}=-\frac{1}{N}\sum_{j=1}^{N}\sum_{i=1,j\neq
i}^{N}v_{j}^{i}logp(i|m_{j}^{t})$ (17)

where $v_{j}\in\left\\{0,1\right\\}^{N}$. If $v_{j}^{i}=1$, it means the
$i$-th image is a reliable neighborhood of the $j$-th image. If $v_{j}^{i}=0$,
the $i$-th image is abandoned.

Final loss The final loss is composed of detection loss and re-id loss, which
is formulated as,

$L=L_{ins}+L_{c}^{t}+L_{c}^{s}+L_{t}^{e}+L_{s}^{e}$ (18)

where $L_{ins}$ is calculated by Eq.9. $L_{c}^{t}$ and $L_{c}^{s}$ are
calculated by Eq.16. $L_{t}^{e}$ and $L_{s}^{e}$ are calculated by Eq.17.

##  3 Experiments

###  3.1 Datasets and Evaluation metrics

Datasets The proposed method is evaluated on two large-scale benchmark
datasets, CUHK-SYSUXiao et al. (2017) and PRWZheng et al. (2017). CUHK-SYSU
consists of 12,490 images captured by real surveillance cameras and 5,694
images from movies and TV shows. The training set comprises 11,206 images and
5,532 query persons, while the test set includes 6,978 images and 2,900 query
persons. The widely utilized PRW dataset comprises a total of 11,816 video
frames captured by 6 cameras and 2,057 queries with 932 identities.

Evaluation metrics In the dataset configuration, annotations for the source
domain dataset are available, while the ground truth boxes and identity
information for the target domain dataset are not accessible. We employ widely
used metrics such as mean average precision(mAP) and cumulative matching
characteristic(CMC) top-1 accuracy for evaluating the re-identification
subtask. For the detection task, average precision(AP) and recall rate are
adopted as the evaluation metrics.

###  3.2 Implementation Details

Our experiments are implemented on NVIDIA GeForce GTX 3090Ti GPU with
PytorchPaszke et al. (2017). FOUS utilize ResNet50He et al. (2016) pre-trained
on ImageNet-1kDeng et al. (2009) as the default backbone network. During the
training process, the dimensions of input images are adjusted to 1500 $\times$
900, and random horizontal flipping is applied for data augmentation. We
employ the Stochastic Gradient Descent(SGD) method to optimize the model, with
a mini-batch size set to 4 and a learning rate of 0.003. The momentum and
weight decay are set to 0.9 and $5\times 10^{-4}$, respectively. For the
k-nearest neighbor algorithm, the number of neighbors is set to 1. When PRW is
chosen as the target domain, we pre-train on the source domain CUHK-SYSU for 8
epochs and subsequently engage in joint training for 10 epochs. Similarly,
when CUHK-SYSU is chosen as the target domain, we pre-train on the source
domain PRW for 2 epochs and subsequently engage in joint training for 14
epochs. However, the similarity between source domain samples is computed only
once.

Table 1: Comparison of mAP($\%$) and rank-1 accuracy($\%$) with the state-of-
the-art on CUHK-SYSU and PRW

Method | CUHK-SYSU | PRW
---|---|---
mAP | top-1 | mAP | top-1
two-stage | DPM Girshick et al. (2015) | - | - | 20.5 | 48.3
MGTS Chen et al. (2018a) | 83.0 | 83.7 | 32.6 | 72.1
CLSA Lan et al. (2018) | 87.2 | 88.5 | 38.7 | 65.0
IGPN Dong et al. (2020) | 90.3 | 91.4 | 47.2 | 87.0
RDLR Han et al. (2019) | 93.0 | 94.2 | 42.9 | 70.2
TCTS Wang et al. (2020a) | 93.9 | 95.1 | 46.8 | 87.5
end-to-end | OIM Xiao et al. (2017) | 75.5 | 78.7 | 21.3 | 49.4
IAN Xiao et al. (2019) | 76.3 | 80.1 | 23.0 | 61.9
NPSM Liu et al. (2017) | 77.9 | 81.2 | 24.2 | 53.1
RCAA Chang et al. (2018) | 79.3 | 81.3 | - | -
CTXG Yan et al. (2019) | 84.1 | 86.5 | 33.4 | 73.6
QEEPS Munjal et al. (2019) | 88.9 | 89.1 | 37.1 | 76.7
APNet Zhong et al. (2020) | 88.9 | 89.3 | 41.9 | 81.4
HOIM Chen et al. (2020a) | 89.7 | 90.8 | 39.8 | 80.4
NAE Chen et al. (2020b) | 91.5 | 92.4 | 43.3 | 80.9
NAE+ Chen et al. (2020b) | 92.1 | 92.9 | 44.0 | 81.1
AlignPS Yan et al. (2021) | 94.0 | 94.5 | 46.1 | 82.1
SeqNet Li and Miao (2021) | 93.8 | 94.6 | 46.7 | 83.4
Ours | FOUS | 78.7 | 80.5 | 35.4 | 80.8

###  3.3 Comparison with State-of-the-Art Methods

As there is currently limited interest in unsupervised pedestrian search
methods, we begin by comparing FOUS with fully supervised methods in Table.1,
encompassing both two-stage and end-to-end methods. For the first time, we
eliminate the need for computationally expensive clustering algorithms in
assigning labels for weakly supervised and unsupervised pedestrian search
tasks. Surprisingly, FOUS outperforms some supervised methods. The
experimental results indicate a substantial gap between FOUS and state-of-the-
art supervised methods. We aspire that our approach may serve as a catalyst
for subsequent explorations in this field.

Table 2: Comparison of mAP($\%$) and rank-1 accuracy($\%$) with the weakly
supervised and unsupervised methods on CUHK-SYSU and PRW

Methods | CUHK-SYSU | PRW
---|---|---
map | top-1 | map | top-1
weakly supervised: |  |  |  |
CGPSYan et al. (2022) | 80.0 | 82.3 | 16.2 | 68.0
R-SiamNetHan et al. (2021) | 86.0 | 87.1 | 21.2 | 73.4
DICLJia et al. (2023) | 87.4 | 88.8 | 35.5 | 80.9
SSLWang et al. (2023) | 87.4 | 88.5 | 30.7 | 80.6
unsupervised: |  |  |  |
DAPSLi et al. (2022) | 77.6 | 79.6 | 34.7 | 80.6
Ours | 78.7 | 80.4 | 35.4 | 80.8

We present a comparison of FOUS with state-of-the-art weakly supervised and
unsupervised methods in Table.2. Surprisingly, FOUS exhibits superior
performance on the PRW dataset compared to all weakly supervised and
unsupervised methods. For the CUHK-SYSU dataset, FOUS surpasses the existing
unsupervised method DAPS by 1% in mAP and 1.2% in top-1, although it has not
yet reached the level of performance achieved by weakly supervised methods.
The reason may be as mentioned in DAPS. Since the number of images and
identities in PRW is significantly less than in CUHK-SYSU, the detection
performance of CUHK-SYSU as the target domain is poor, further leading to
lower accuracy in the search task. The proposed attention-based domain
alignment module enhances the quality of proposals, further bolstering the
performance of detection in FOUS. Consequently, FOUS achieves a balance
between reducing computational complexity, accelerating processing speed, and
maintaining stable accuracy.

(a) $N_{t}$

(b) eopch

Figure 5: 5(a) represents the influence of varying the number of random prototypes on the map and top-1 metrics on the CUHK-SYSU dataset. 5(b) illustrates the impact of different pre-training rounds on two datasets.  Table 3: Validating the effectiveness of different modules on CUHK and PRW datasets. AM: Attention Mechanism. TDA: Task-sensitive Domain Alignment. LFT: Label-flexible Training AM | TDA | LFT | CUHK-SYSU | PRW
---|---|---|---|---
map | top-1 | map | top-1
|  |  | 50.4 | 52.8 | 26.7 | 74.6
$\checkmark$ | $\checkmark$ |  | 68.1 | 71.6 | 33.2 | 80.4
| $\checkmark$ | $\checkmark$ | 64.3 | 66.9 | 28.0 | 77.1
$\checkmark$ |  | $\checkmark$ | 72.3 | 75.7 | 32.9 | 80.2
$\checkmark$ | $\checkmark$ | $\checkmark$ | 78.7 | 80.4 | 35.4 | 80.8

###  3.4 Ablation Study

In this subsection, we perform ablation experiments to validate the
significance of each component in FOUS. In Table.3, we compare the combination
of baseline methods with different proposed modules and report the results on
both CUHK-SYSU and PRW datasets. The baseline model achieves 50.4% mAP and
52.8% Top-1 accuracy on the CUHK-SYSU dataset. FOUS implemented three
components and achieved a 28.2% mAP and 28.0% top-1 improvement over the
baseline. We find that if only use the prototype-guided labeling method to
generate coarse label groups without utilizing the label-flexible network to
refine the labels, the accuracy will decrease significantly by 10% on CUHK-
SYSU and 7% on PRW. If the attention mechanism is abandoned to enhance
proposals, the accuracy drops even further to 14.3% on CUHK-SYSU. Furthermore,
neglecting the domain difference between the two subtasks leads to a reduction
in accuracy by 6.3% on CUHK. Hence, the three components we proposed
effectively enhance the performance and generalization ability of the model.

Furthermore, to verify the effectiveness of the designed attention module, we
compared it with state-of-the-art attention methods. As shown in Table.5, our
proposed attention module outperforms significantly, achieving an average
accuracy 1.8 higher than CBAM, ECAM, SGEM, and GCT methods on CUHK-SYSU. We
can confidently infer that global information aggregation and interaction with
proposals are beneficial for model learning.

Last but not least, we compare with state-of-the-art weakly supervised and
unsupervised methods in terms of network parameter amount and computational
effort. Table.4 presents the comparative results concerning computational
requirements, parameter count, training time per iteration, and accuracy
during the process. As FOUS computes the similarity of source domain samples
only once, it reduces computational costs by nearly half compared to DAPS,
which utilizes clustering algorithms. This reduction is achieved at a minor
cost in terms of parameters. Furthermore, training time is relatively reduced
by 25%, and there is a remarkable improvement in accuracy rather than a
decline. We have abandoned the complex clustering method and, instead,
proposed a prototype correlation labeling algorithm, resulting in a
significant reduction in the number of parameters and computational workload.

###  3.5 Parameter analysis

Setting several parameters in FOUS may affect the performance of the model.
Therefore, we evaluate these parameters in our paper, such as the number of
random-prototype vectors and the pre-trained epochs during the training stage.
The results on CUHK-SYSU and PRW under different settings are shown in Fig.5.

Analysis of the random selected prototype number As shown in Fig.5(a), we vary
the number of random prototype vectors $N_{t}$ to 400, 600, 800, 1000, and
1200 on CUHK-SYSU, respectively. And $N_{t}$ is set to 100, 200, 300, 400, 500
on PRW, respectively. In detail, maintaining the parameters of batch size and
pretraining rounds unchanged, as illustrated in Figure 3, the model
performance varies with the different settings of $N_{t}$. This underscores
the significance of randomly selecting the number of prototypes for generating
pseudo-labels. What’s more, Fig.5(a) also shows that $N_{t}$ = 600 and $N_{t}$
= 300 can achieve the best accuracy on CUHK-SYSU and PRW, respectively.

Analysis of the pre-trained epochs In this part, the paper explores the
relations between the pre-trained epochs and the whole model performance. As
observed in Fig.5(b), when CUHK-SYSU is embraced as the target domain, the
performance peaks at a pre-trained epoch of 10, while with epoch = 4 for PRW.
Our analysis of this result is that in smaller source domains, even limited
additional target information may contribute to cross-domain generalization.
In contrast, for larger source datasets, unreliable target suggestions may
have a negative impact on mitigating domain differences.

Table 4: Comparing the network parameters and computational requirements with
weakly supervised and unsupervised methods on PRW.

Methods | GFLOPS | PRARMS | Time | map | top-1
---|---|---|---|---|---
CGPS | 281 | 49.2M | 1.1h | 16.2 | 68.0
R-SiamNet | 320 | 52.5M | 1h | 21.4 | 75.2
DAPS | 628 | 53.4M | 1.6h | 34.7 | 80.6
Ours | 388 | 53.6M | 1.2h | 35.4 | 80.8

Table 5: Validating the effectiveness of different attention mechanism on CUHK
and PRW datasets.

Attention Models | CUHK-SYSU | PRW
---|---|---
map | top-1 | map | top-1
CBAMWoo et al. (2018) | 77.2 | 79.0 | 33.6 | 79.4
ECAMWang et al. (2020b) | 76.2 | 78.1 | 30.1 | 78.6
SGEMLi et al. (2019) | 77.0 | 78.7 | 32.3 | 80.1
GCTRuan et al. (2021) | 76.8 | 77.9 | 32.6 | 80.2
Ours | 78.7 | 80.4 | 35.4 | 80.8

##  4 Conclusion

In this paper, we present the FOUS framework designed for the domain adaptive
pedestrian search task. FOUS introduces two components, one is a prototype-
guided labeling algorithm, which replaces the commonly used complex clustering
algorithms in this domain, significantly reducing computational costs at the
expense of accuracy. The other is an attention-based domain alignment module,
alleviating the impact of low-quality candidate boxes on re-identification
tasks in unsupervised detection. Together, these components complement each
other, reducing computational overhead while enhancing the model’s
performance. Judging from the results, experiments on two foundational
datasets demonstrate the performance of FOUS and the effectiveness of the
introduced modules and achieve state-of-the-art performance.

## References

  * Chang et al. [2018] Xiaojun Chang, Po-Yao Huang, Yi-Dong Shen, Xiaodan Liang, Yi Yang, and Alexander G Hauptmann.  Rcaa: Relational context-aware agents for person search.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 84–100, 2018.
  * Chen et al. [2018a] Di Chen, Shanshan Zhang, Wanli Ouyang, Jian Yang, and Ying Tai.  Person search via a mask-guided two-stream cnn model.  In Proceedings of the european conference on computer vision (ECCV), pages 734–750, 2018.
  * Chen et al. [2018b] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Domain adaptive faster r-cnn for object detection in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3339–3348, 2018.
  * Chen et al. [2020a] Di Chen, Shanshan Zhang, Wanli Ouyang, Jian Yang, and Bernt Schiele.  Hierarchical online instance matching for person search.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10518–10525, 2020.
  * Chen et al. [2020b] Di Chen, Shanshan Zhang, Jian Yang, and Bernt Schiele.  Norm-aware embedding for efficient person search.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12615–12624, 2020.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * Ding et al. [2020] Yuhang Ding, Hehe Fan, Mingliang Xu, and Yi Yang.  Adaptive exploration for unsupervised person re-identification.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 16(1):1–19, 2020.
  * Dong et al. [2020] Wenkai Dong, Zhaoxiang Zhang, Chunfeng Song, and Tieniu Tan.  Instance guided proposal network for person search.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2585–2594, 2020.
  * Girshick et al. [2015] Ross Girshick, Forrest Iandola, Trevor Darrell, and Jitendra Malik.  Deformable part models are convolutional neural networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 437–446, 2015.
  * Han et al. [2019] Chuchu Han, Jiacheng Ye, Yunshan Zhong, Xin Tan, Chi Zhang, Changxin Gao, and Nong Sang.  Re-id driven localization refinement for person search.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9814–9823, 2019.
  * Han et al. [2021] Chuchu Han, Kai Su, Dongdong Yu, Zehuan Yuan, Changxin Gao, Nong Sang, Yi Yang, and Changhu Wang.  Weakly supervised person search with region siamese networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12006–12015, 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * Jia et al. [2023] Chengyou Jia, Minnan Luo, Caixia Yan, Linchao Zhu, Xiaojun Chang, and Qinghua Zheng.  Collaborative contrastive refining for weakly supervised person search.  IEEE Transactions on Image Processing, 2023.
  * Khodabandeh et al. [2019] Mehran Khodabandeh, Arash Vahdat, Mani Ranjbar, and William G Macready.  A robust learning approach to domain adaptive object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 480–490, 2019.
  * Lan et al. [2018] Xu Lan, Xiatian Zhu, and Shaogang Gong.  Person search by multi-scale matching.  In Proceedings of the European conference on computer vision (ECCV), pages 536–552, 2018.
  * Li and Miao [2021] Zhengjia Li and Duoqian Miao.  Sequential end-to-end network for efficient person search.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2011–2019, 2021.
  * Li et al. [2019] Xiang Li, Xiaolin Hu, and Jian Yang.  Spatial group-wise enhance: Improving semantic feature learning in convolutional networks.  arXiv preprint arXiv:1905.09646, 2019.
  * Li et al. [2022] Junjie Li, Yichao Yan, Guanshuo Wang, Fufu Yu, Qiong Jia, and Shouhong Ding.  Domain adaptive person search.  In European Conference on Computer Vision, pages 302–318. Springer, 2022.
  * Liu et al. [2017] Hao Liu, Jiashi Feng, Zequn Jie, Karlekar Jayashree, Bo Zhao, Meibin Qi, Jianguo Jiang, and Shuicheng Yan.  Neural person search machines.  In Proceedings of the IEEE International Conference on Computer Vision, pages 493–501, 2017.
  * Munjal et al. [2019] Bharti Munjal, Sikandar Amin, Federico Tombari, and Fabio Galasso.  Query-guided end-to-end person search.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 811–820, 2019.
  * Paszke et al. [2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * Peng et al. [2020] Jinjia Peng, Yang Wang, Huibing Wang, Zhao Zhang, Xianping Fu, and Meng Wang.  Unsupervised vehicle re-identification with progressive adaptation.  In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 913–919, 7 2020.
  * Ruan et al. [2021] Dongsheng Ruan, Daiyin Wang, Yuan Zheng, Nenggan Zheng, and Min Zheng.  Gaussian context transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15129–15138, 2021.
  * Vibashan et al. [2023] VS Vibashan, Poojan Oza, and Vishal M Patel.  Instance relation graph guided source-free domain adaptive object detection.  In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3520–3530. IEEE, 2023.
  * Wang et al. [2020a] Cheng Wang, Bingpeng Ma, Hong Chang, Shiguang Shan, and Xilin Chen.  Tcts: A task-consistent two-stage framework for person search.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11952–11961, 2020.
  * Wang et al. [2020b] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu.  Eca-net: Efficient channel attention for deep convolutional neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11534–11542, 2020.
  * Wang et al. [2022] Yang Wang, Jinjia Peng, Huibing Wang, and Meng Wang.  Progressive learning with multi-scale attention network for cross-domain vehicle re-identification.  Science China Information Sciences, 65(6):160103, 2022.
  * Wang et al. [2023] Benzhi Wang, Yang Yang, Jinlin Wu, Guo-jun Qi, and Zhen Lei.  Self-similarity driven scale-invariant learning for weakly supervised person search.  arXiv preprint arXiv:2302.12986, 2023.
  * Wang et al. [2024] Yang Wang, Biao Qian, Haipeng Liu, Yong Rui, and Meng Wang.  Unpacking the gap box against data-free knowledge distillation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024\.
  * Wang [2021] Yang Wang.  Survey on deep multi-modal data analytics: Collaboration, rivalry, and fusion.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 17(1s):1–25, 2021.
  * Woo et al. [2018] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * Xiao et al. [2017] Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang.  Joint detection and identification feature learning for person search.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3415–3424, 2017.
  * Xiao et al. [2019] Jimin Xiao, Yanchun Xie, Tammam Tillo, Kaizhu Huang, Yunchao Wei, and Jiashi Feng.  Ian: the individual aggregation network for person search.  Pattern Recognition, 87:332–340, 2019.
  * Xu et al. [2020] Chang-Dong Xu, Xing-Ran Zhao, Xin Jin, and Xiu-Shen Wei.  Exploring categorical regularization for domain adaptive object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11724–11733, 2020.
  * Yan et al. [2019] Yichao Yan, Qiang Zhang, Bingbing Ni, Wendong Zhang, Minghao Xu, and Xiaokang Yang.  Learning context graph for person search.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2158–2167, 2019.
  * Yan et al. [2021] Yichao Yan, Jinpeng Li, Jie Qin, Song Bai, Shengcai Liao, Li Liu, Fan Zhu, and Ling Shao.  Anchor-free person search.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7690–7699, 2021.
  * Yan et al. [2022] Yichao Yan, Jinpeng Li, Shengcai Liao, Jie Qin, Bingbing Ni, Ke Lu, and Xiaokang Yang.  Exploring visual context for weakly supervised person search.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3027–3035, 2022.
  * You et al. [2019] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Universal domain adaptation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2720–2729, 2019.
  * Zhang et al. [2023] Jingyi Zhang, Jiaxing Huang, Zhipeng Luo, Gongjie Zhang, Xiaoqin Zhang, and Shijian Lu.  Da-detr: Domain adaptive detection transformer with information fusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23787–23798, 2023.
  * Zheng et al. [2017] Liang Zheng, Hengheng Zhang, Shaoyan Sun, Manmohan Chandraker, Yi Yang, and Qi Tian.  Person re-identification in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1367–1376, 2017.
  * Zhong et al. [2020] Yingji Zhong, Xiaoyu Wang, and Shiliang Zhang.  Robust partial matching for person search in the wild.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6827–6835, 2020.


RL

    reinforcement learning
MDP

    Markov decision process
PPO

    proximal policy optimization
SAC

    soft actor-critic
HRC

    human-robot collaboration
DoF

    degree of freedom
ISS

    invariably safe state
SSM

    speed and separation monitoring
PFL

    power and force limiting
RSI

    reference state initialization
SIR

    state-based imitation reward
AIR

    action-based imitation reward
PID

    proportional–integral–derivative
PDF

    probability density function

# Human-Robot Gym: Benchmarking Reinforcement Learning in Human-Robot
Collaboration

Jakob Thumm, Felix Trost, and Matthias Althoff The authors are with the
Department of Informatics, Technical University of Munich, 85748 Garching,
Germany<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>work has
been submitted to the IEEE for possible publication. Copyright may be
transferred without notice, after which this version may no longer be
accessible

###### Abstract

Deep reinforcement learning (RL) has shown promising results in robot motion
planning with first attempts in human-robot collaboration (HRC). However, a
fair comparison of RL approaches in HRC under the constraint of guaranteed
safety is yet to be made. We, therefore, present human-robot gym, a benchmark
for safe RL in HRC. Our benchmark provides eight challenging, realistic HRC
tasks in a modular simulation framework. Most importantly, human-robot gym
includes a safety shield that provably guarantees human safety. We are,
thereby, the first to provide a benchmark to train RL agents that adhere to
the safety specifications of real-world HRC. This bridges a critical gap
between theoretic RL research and its real-world deployment. Our evaluation of
six environments led to three key results: (a) the diverse nature of the tasks
offered by human-robot gym creates a challenging benchmark for state-of-the-
art RL methods, (b) incorporating expert knowledge in the RL training in the
form of an action-based reward can outperform the expert, and (c) our agents
negligibly overfit to training data.

##  I Introduction

Recent advancements in deep reinforcement learning (RL) are promising for
solving intricate decision-making processes [1] and complex manipulation tasks
[2]. These capabilities are pivotal for human-robot collaboration (HRC), given
that robotic systems must act in environments featuring highly nonlinear human
dynamics. Despite the promising outlook, the few works on RL in HRC confine
themselves to narrow task domains [3]. Two primary challenges impeding the
widespread integration of RL in HRC are safety concerns and the diversity of
tasks. The assurance of safety for RL agents operating within human-centric
environments is a hurdle as agents generate potentially unpredictable actions,
posing substantial risks to human collaborators. Current HRC benchmarks
circumvent these safety concerns by focusing primarily on interacting with
stationary humans.

In this paper, we propose human-robot gym111human-robot gym is available at
https://github.com/TUMcps/human-robot-gym, an HRC benchmark that comes with a
broad range of tasks, including object inspection, handovers, and
collaborative manipulation, while ensuring safe robot behavior by integrating
SaRA shield [4], a tool for provably safe RL in HRC. With its set of
challenging HRC tasks, human-robot gym enables training RL agents to
collaborate with humans in a safe manner, which is not possible with other
benchmarks. We designed our benchmark modularly so that users can easily
define new tasks, adjust existing tasks, and adapt training preferences with
simple changes in the configuration files without changing any code.
Additionally, we provide examples for adding new objects, custom controllers,
sensors, and human animations to our environments. Human-robot gym tracks all
relevant performance and safety metrics to ensure comparability between
solution proposals.

Together with our benchmark, we provide helpful features that lower the entry
barrier into the field of RL in HRC. First, we provide safety mechanisms for
both static and dynamic objects. Since RL agents should only be deployed on a
real robot in a human environment with appropriate safety measures,
integrating human safety into our benchmark enables a realistic training
environment. Additionally, we include an optional stable-baselines 3 [5]
integration so users can conveniently configure and train state-of-the-art RL
agents in human-robot gym. All other RL algorithms that support the OpenAI gym
standard are also compatible with human-robot gym. Lastly, we hand-crafted
expert policies for every task to compare the performance of the learned
policies with them. The expert policies can also be used to gather data for
imitation-based RL methods, as described in [6, 7, 8]. We show the benefit of
this expert data in our experimental evaluation, which will also serve as a
baseline for safe RL in HRC. All our training and evaluation results are
reproducible with a single command.

Reach

Pick and place

Object inspection

Lifting

Robot-human handover

Human-robot handover

Collaborative stacking

Collaborative hammering

Figure 1: Human-robot gym presents eight challenging HRC tasks.

To summarize, human-robot gym features:

  * •

Eight pre-defined scenarios, see Fig. 1, with varying difficulty, each with a
set of real-world human animations.

  * •

Provably safe RL using SaRA shield.

  * •

Seven robots are available for training (Panda, Sawyer, IIWA, Jaco, Kinova3,
UR5e, and Schunk).

  * •

High fidelity simulation based on MuJoCo [9].

  * •

Highly configurable and expandable scenarios.

  * •

Supports joint space and workspace actions.

  * •

Static obstacle and self-collision prevention.

  * •

Pre-defined expert policies for gathering imitation data and performance
comparison.

This article is structured as follows: Sec. II introduces previous work in RL
for HRC, compares human-robot gym to other related benchmarks in the field,
and gives a short overview of imitation learning approaches. Sec. III presents
our benchmark in detail. We then present additional tools supporting users to
solve human-robot gym tasks in Sec. IV. Sec. V evaluates our benchmark
experimentally and discusses the results. Finally, we conclude this work in
Sec. VI.

##  II Related Work

Semeraro et al. [3] summarize recent efforts in machine learning for HRC. They
identify four typical HRC applications: collaborative assembly [10, 11],
object handover [12, 13], object handling [14, 15], and collaborative
manufacturing [16].

Recent developments in RL evoke the need for comparable benchmark environments
in various applications. One of the most used benchmarks for robotic
manipulation is robosuite[17], which offers a set of diverse robot models,
realistic sensor and actuator models, simple task generation, and a high-
fidelity simulation using MuJoCo [9]. Further notable manipulation benchmarks
include Orbit [18], which focuses on photorealism; OmniGibson, which also
provides a benchmark of 1000 everyday robotic tasks [19]; and meta-world [20]
for meta RL research.

None of the benchmarks mentioned above include humans in the simulation. There
are, however, some benchmarks that provide limited human capabilities with a
specific research focus. First, the robot interaction in virtual reality [21]
and SIGVerse [22] benchmarks include real humans in real-time teleoperation
through virtual reality setups. Unfortunately, this approach is unsuitable for
training an RL agent from scratch due to high training times. Closest to our
work are AssistiveGym [23] and RCareWorld [24]. These benchmarks provide
simulation environments for ambulant caregiving tasks. RCareWorld provides a
large set of assistive tasks using a realistic human model and a choice of
robot manipulators. However, both environments focus on tasks where the human
is primarily static or only features small, limited motions. Comparably, our
work focuses on collaborative tasks, where the human and the robot play an
active role, and the human motions are thus complex. Furthermore, one primary
focus of human-robot gym is human safety, which RCareWorld and AssistiveGym
only discuss superficially. Also closely related to our work is HandoverSim
[25], which investigates the handover of diverse objects from humans to
robots. Here, prerecorded motion-capturing clips steer the human hand.
However, these animations only capture the hand picking up objects and
presenting them to the robot. From that point onward, the hand remains
motionless [25]. Compared to our work, they do not supply animation data while
the handover is ongoing, limiting their interactivity. Our benchmark further
includes a broader selection of tasks and considers safety concerns.

We utilize learning from experts [7] to provide the first results on our
benchmark tasks. Currently, we mainly rely on two techniques: reference state
initialization, which lets the agent start at a random point of an expert
trajectory [26], and state-based imitation reward, which additionally rewards
the agent for being close to the expert trajectory [6]. We explicitly decided
against behavior cloning techniques [8] as they merely copy the expert
behavior and often fail to generalize to the task objective [7].

##  III Benchmark

Figure 2:  A typical workflow of an RL cycle in human-robot gym. Optional
elements are depicted with dashed borders, and the inner loop of the
environment step is executed $L$ times, e.g., $L=20$. In this example, the
agent outputs an action in Cartesian space for the end effector position,
which is converted into a joint position action using the inverse kinematics
wrapper. The static collision prevention checks the desired target goal joint
position for self-collision, and static environment collision and alters the
action if necessary. The shield calculates the next safe joint position, which
the joint position controller then converts into the desired robot joint
torque. The robot and human motion is simulated in MuJoCo.

We base human-robot gym on robosuite [17], which already provides adjustable
robot controllers, a high-fidelity simulation environment with MuJoCo. We
support seven different robot models: Panda, Sawyer, IIWA, Jaco, Kinova3,
UR5e, and Schunk. Primarily, our environments introduce the functionality to
interact with a human entity, define tasks with complex collaboration
objectives, and evaluate human safety. All our tasks are definable through
modular configuration files with the Hydra framework [27], which makes human-
robot gym easily configurable and extendable. Each task consists of a main
configuration file, consisting of pointers to configuration files for
environment definition, robot specifications, environment wrapper settings,
expert policy descriptions, training parameters, and RL algorithm
hyperparameters. Hydra also allows users to run multiple training runs with
changing parameters with one command, e.g., training on multiple random seeds.
In the following subsections, we describe our environments, the typical
workflow of human-robot gym, and its elements in more detail.

###  III-A Environments

Table I: Environment characteristics Environment |  HRC category1 |  Safety mode2 | Manipulation | Time-horizon | Dynamics | Reward | Delayed reward | No. of animations
---|---|---|---|---|---|---|---|---
Reach | coexistence | SSM | easy | easy | easy | dense | no | 12
Pick and place | coexistence | SSM | medium | medium | easy | sparse | no | 12
Object inspection | object handling | SSM | medium | medium | medium | sparse | yes | 8
Collaborative lifting | object handling | SSM | medium | medium | medium | dense | no | 9
Robot-human handover | object handover | PFL | medium | medium | hard | sparse | yes | 15
Human-robot handover | object handover | PFL | hard | medium | hard | sparse | yes | 11
Collaborative hammering | object manufacturing | SSM | hard | medium | hard | sparse | yes | 11
Collaborative stacking | object assembly | SSM | hard | hard | hard | sparse | yes | 8
1from [3], 2SSM: speed and separation monitoring, PFL: power and force
limiting

Human-robot gym features tasks that reflect the HRC categories introduced in
[3]. Additionally, we selected two typical coexistence tasks, reach as well as
pick and place. Table I displays the default settings of each task, which we
also use in our experiments, and a subjective estimate of how challenging each
environment is regarding manipulation, length of the time horizon, and human
dynamics. The reward in our environments can be sparse, e.g., indicating
whether an object is at the target position, and dense, e.g., proportional to
the Euclidean distance to the goal. Furthermore, environments can have a
delayed sparse reward signal, which should mimic a realistic HRC environment,
where the agent receives the task fulfillment reward shortly after the action
that completes the task. An example of a delayed reward is when a handover was
successful, but the human needs a short time to approve the execution. The
reward delay serves as an additional challenge for the RL agents. We present
the modes of the tool SaRA shield in Section IV-A.

###  III-B Typical workflow

Fig. 2 displays a typical workflow of an RL cycle in human-robot gym. The
actions of the RL agent can be joint space or workspace actions. The workspace
actions can include the end effector orientation in $\mathrm{SO}(3)$. However,
in our experiments, we only use the desired positional difference of the end
effector in Cartesian space ${\bm{a}}_{\text{EEF}}={\bm{p}}_{\text{EEF,
desired}}-{\bm{p}}_{\text{EEF}}$ as actions, where the gripper is pointing
downwards. This simplification to a four-dimensional action space (three
positional actions and a gripper action) is common in literature [2, 28, 29].
If ${\bm{a}}_{\text{EEF}}$ is selected, the inverse kinematics wrapper
determines ${\bm{p}}_{\text{joint, desired}}$ from ${\bm{p}}_{\text{EEF,
desired}}$ and outputs the joint action. The RL action might violate safety
constraints. Users can, therefore, implement safety functionalities as part of
the outer RL loop or the inner environment loop. We present how our additional
tools use both variants to prevent collisions with static obstacles and
guarantee human safety in Sec. IV. The step function of our environment
executes its inner loop $L$ times. Every iteration of the inner loop runs the
optional inner safety function, the robot controller, one fixed step of the
MuJoCo simulation, and the human measurement. After executing the action, the
environment outputs an observation $s$ and a reward $r$ to the agent.

###  III-C Human simulation

Our benchmark moves the human using motion capture files obtained from a Vicon
tracking system. All movements are recorded specifically for the defined tasks
and include task-relevant objects in the scene, ensuring realistic behavior. A
limitation of using recordings are instances where the recording must be
paused until the robot initiates a specific event, e.g., in a handover task.
Previous works show an unnatural human behavior in these cases. To address
this limitation, we incorporate idle movements representing the human waiting
for the event trigger. For each recording, keyframes can designate the start
and end of an idle phase. Once reached, the movement remains idle until an
event trigger $\sigma_{E}$ is detected, at which point it progresses to the
successive movement. The boolean trigger $\sigma_{E}$ is true when the robot
achieves a task-specific sub-goal and thereafter, e.g., handover an object.
Instead of simply looping the idle phase, which would lead to jumps in the
movement, we scale the animation time with a set of $D$ superimposing sine-
functions:

$\displaystyle t_{A}=\begin{cases}&t,\text{ if }t\leq t_{I}\lor\sigma_{E}\\\
&t_{I}+\sum\limits_{i=1}^{D}\upsilon_{i}\sin\left(\left(t-t_{I}\right)\omega_{i}\right),\text{
otherwise}\,,\end{cases}$ (1)

where $\upsilon_{i}$ and $\omega_{i}$ define the amplitude and frequency of
the $i$-th sine-function during idling respectively and both are randomized at
the start of each episode. The animation time can also reverse in the idling
phase. The recordings to replay are randomly selected, and their starting
position and orientation are slightly randomized at the start of each episode
to avoid overfitting.

###  III-D Observation

Table II: Observation elements | Element | Variable | Relative | Distance
---|---|---|---|---
Robot | Joint angle | ${\bm{q}}$ | – | –
Joint velocity | $\dot{{\bm{q}}}$ | – | –
EEF aperture | $\varphi$ | – | –
EEF position | ${\bm{p}}_{\text{EEF}}$ | – | –
EEF orientation | ${\bm{o}}_{\text{EEF}}$ | – | –
Task | Object position | ${\bm{p}}_{\text{obj},a}$ | ${\bm{p}}_{\text{obj},a}-{\bm{p}}_{\text{EEF}}$ | $\|{\bm{p}}_{\text{obj},a}-{\bm{p}}_{\text{EEF}}\|_{2}$
Object orientation | ${\bm{o}}_{\text{obj},a}$ | ${\bm{o}}_{\text{obj},a}-{\bm{o}}_{\text{EEF}}$ | –
Obstacle position | ${\bm{p}}_{\text{obs},b}$ | ${\bm{p}}_{\text{obs},b}-{\bm{p}}_{\text{EEF}}$ | $\|{\bm{p}}_{\text{obs},b}-{\bm{p}}_{\text{EEF}}\|_{2}$
Obstacle orientation | ${\bm{o}}_{\text{obs},b}$ | ${\bm{o}}_{\text{obs},b}-{\bm{o}}_{\text{EEF}}$ | –
Target position | ${\bm{p}}_{\text{target},c}$ | ${\bm{p}}_{\text{target},c}-{\bm{p}}_{\text{EEF}}$ | $\|{\bm{p}}_{\text{target},c}-{\bm{p}}_{\text{EEF}}\|_{2}$
Target joint angle | ${\bm{q}}_{\text{target},c}$ | ${\bm{q}}_{\text{target},c}-{\bm{q}}$ | $\|{\bm{q}}_{\text{target},c}-{\bm{q}}\|_{2}$
Object gripped | $\sigma_{\text{grip}}$ | – | –
Object at target | $\sigma_{\text{target}}$ | – | –
Static collision | $\sigma_{\text{col, stat}}$ | – | –
Human | Body position | ${\bm{p}}_{\text{body},d}$ | ${\bm{p}}_{\text{body},d}-{\bm{p}}_{\text{EEF}}$ | $\|{\bm{p}}_{\text{body},d}-{\bm{p}}_{\text{EEF}}\|_{2}$
Safe human contact | $\sigma_{\text{contact}}$ | – | –
Critical human contact | $\sigma_{\text{crit}}$ | – | –

Human-robot-gym features observations containing task-specific, robotic, and
human elements, as shown in Table II. Objects, obstacles, targets, and human
bodies are observable in absolute position ${\bm{p}}$, position difference to
end effector ${\bm{p}}-{\bm{p}}_{\text{EEF}}$, or by the distance to the end
effector $\|{\bm{p}}-{\bm{p}}_{\text{EEF}}\|_{2}$. Additionally, the
orientation ${\bm{o}}$ is given through quaternions. The task-specific
elements in Table II include those necessary to fulfill the task, i.e.,
${\bm{p}}_{\text{obj},a},a=1,\dots,A$, ${\bm{p}}_{\text{obs},b},b=1,\dots,B$,
and ${\bm{p}}_{\text{target},c},c=1,\dots,C$, with $A$ objects, $B$ obstacles,
and $C$ target positions. The robot information contains its joint positions
and velocities as well as the end effector position, orientation, and
aperture. The agent can further get information about the $D$ human body
positions ${\bm{p}}_{\text{body},d},d=1,\dots,D$. In our experiments, we found
that reducing the number of elements in the observation, e.g., only providing
measurements of the human hand positions instead of the entire human model, is
beneficial for training performance. To emulate real-world sensors, users can
optionally add noise sampled from a compact set and delays to all
measurements, further closing the gap between simulation and reality. In
addition to the physical measurements, the user can define cameras that
observe the scene and learn with vision inputs.

##  IV Supporting tools

This section describes additional tools included in human-robot gym to provide
safety and RL training functionality.

###  IV-A Safety tools

We can prevent static and self-collisions in the outer RL loop by performing
collision checks of the RL action using pinocchio [30]. If the action is
unsafe, we sample random actions until we find a safe action.

Guaranteeing human safety in the outer RL loop is challenging. As the time
horizon of RL actions is relatively long, e.g., $200\text{\,}\mathrm{ms}$,
checking safety only once before execution would lead to a very restrictive
safety behavior [31]. Therefore, we ensure human safety in the inner
environment loop. We provide the tool SaRA shield introduced for robotic
manipulators in [4] and generalized to arbitrary robotic systems in [31].
First, SaRA shield translates each RL action into an intended trajectory. In
the subsequent period of an RL action, the shield is executed $L$ times. In
each timestep, the shield computes a failsafe trajectory, which guides the
robot into an invariably safe state (ISS). As defined in [4], an ISS in
manipulation is a condition where the robot completely stops in compliance
with the ISO 10218-1 2021 regulations [32]. Next, the shield constructs a
shielded trajectory combining one timestep from the planned intended
trajectory with the failsafe trajectory. SaRA shield validates these shielded
trajectories through set-based reachability analysis of the human and robot.
For this, the shield receives the position and velocity of human body parts as
measurements from the simulation. We assure safety indefinitely, provided that
the initial state of the system is an ISS, by only executing the step from the
intended trajectory when the shielded trajectory is confirmed safe [4]. In the
event of a failed safety verification, the robot follows the most recently
validated failsafe trajectory, guaranteeing continued safe operation. Finally,
SaRA shield outputs the desired robot joint position for the next timestep to
follow the verified trajectory. We then use a proportional–integral–derivative
(PID) controller to calculate the desired robot joint torque.

The default mode of SaRA shield is speed and separation monitoring, which
stops the robot before any collision with the human occurs. This is too
restrictive for close interaction tasks, such as handovers, as the robot must
come into contact with the human. Therefore, we include a power and force
limiting mode in the tool SaRA shield that decelerates the robot to a safe
Cartesian velocity of $5\text{\,}\mathrm{mm}\text{\,}{\mathrm{s}}^{-1}$ before
any human contact, as proposed in [33, Def. 3]. Thereby, our power and force
limiting mode ensures painless contact in accordance with ISO 10218-1 2021
[32]. As in the speed and separation monitoring mode, SaRA shield only slows
down the robot if our reachability-based verification detects a potential
collision. Otherwise, the robot is allowed to operate at full speed. We
further plan to include a conformant impedance controller, as proposed in
[34], in SaRA shield in the future.

###  IV-B Tools for training

To provide a perspective on the performance of RL agents in our environments,
we provide both expert and RL policies with our tasks. In this work, we
consider an RL agent that learns on a Markov decision process described by the
tuple $\left({\mathbb{S}},{\mathbb{A}},T,r,{\mathbb{S}}_{0},\gamma\right)$ in
both continuous or discrete action spaces ${\mathbb{A}}$ and continuous state
spaces ${\mathbb{S}}$ with a set of initial states ${\mathbb{S}}_{0}$. Here,
$T({\bm{s}}^{\prime}|{\bm{s}},{\bm{a}})$ is the transition function, which
denotes the probability density function of transitioning from state
${\bm{s}}$ to ${\bm{s}}^{\prime}$ when action ${\bm{a}}$ is taken. The agent
receives a reward determined by the function
$r:{\mathbb{S}}\times{\mathbb{A}}\times{\mathbb{S}}\rightarrow\mathbb{R}$ from
the environment. Lastly, we consider a discount factor $\gamma\in[0,1]$ to
adjust the relevance of future rewards. RL aims to learn an optimal policy
$\pi^{\star}({\bm{a}}|{\bm{s}})$ that maximizes the expected return
$R=\sum_{k=0}^{K}\gamma^{k}r({\bm{s}}_{k},{\bm{a}}_{k},{\bm{s}}_{k+1})$ when
starting from an initial state ${\bm{s}}_{0}\in{\mathbb{S}}_{0}$ and following
$\pi^{\star}({\bm{a}}|{\bm{s}})$ until termination at $k=K$ [35].

###  IV-C Pre-defined experts

We define a deterministic expert policy $\pi_{\mathrm{e}}({\bm{a}}|{\bm{s}})$
for each task to gather imitation data and compare performance. The experts
are hand-crafted and follow a proportional control law with heuristics based
on human expertise strategy, as described in full detail in the human-robot
gym documentation.

To achieve diversity in our expert data, we add a noise term to the expert
action, resulting in the noisy expert

$\displaystyle\tilde{\pi}_{\mathrm{e}}({\bm{a}}|{\bm{s}},t)=\pi_{\mathrm{e}}({\bm{a}}|{\bm{s}})*f_{{\mathbf{n}}}({\mathbf{n}}|t)\,,$
(2)

where $*$ denotes the convolution of probability distributions, and
$f_{{\mathbf{n}}}({\mathbf{n}}|t)$ is the probability density function of the
noise. In our experiments, we use a noise that stems from an Ornstein-
Uhlenbeck process [36], which is a mean-reverting process. We model
${\mathbf{n}}$ to be a vector of independent random variables
${\textnormal{n}}_{i}$ with zero mean, whose probability density functions
$f_{{\textnormal{n}}_{i}}$ are normal distributions [37, (7.2.14)] with

$\displaystyle
f_{{\textnormal{n}}_{i}}({\textnormal{n}}_{i}|t)=\mathcal{N}\left(\hat{{\textnormal{n}}}_{i}e^{-\alpha_{i}t},\frac{\beta_{i}^{2}}{2\alpha}\left(1-e^{-2\alpha_{i}t}\right)\right)\,,$
(3)

where $\alpha_{i}$ and $\beta_{i}$ parameterize the Ornstein-Uhlenbeck
processes, and $\hat{{\textnormal{n}}}_{i}$ is the noise applied at $t=0$. We
can sample an expert trajectory
$\chi=\left({\bm{s}}_{0},\dots,{\bm{s}}_{K}\right)$ with a Monte Carlo
simulation, where we start in ${\bm{s}}_{0}\in{\mathbb{S}}_{0}$, and
subsequently follow ${\bm{s}}_{k+1}\sim
T({\bm{s}}_{k+1}|{\bm{s}}_{k},{\bm{a}}_{k})$ with
${\bm{a}}_{k}\sim\tilde{\pi}_{\mathrm{e}}({\bm{a}}_{k}|{\bm{s}}_{k},t_{k})$
for $k=0,\dots,K-1$. For each task in human-robot gym, we provide the experts
$\pi_{\mathrm{e}}({\bm{a}}|{\bm{s}})$ and
$\tilde{\pi}_{\mathrm{e}}({\bm{a}}|{\bm{s}},t)$ together with a set of $M$
expert trajectories ${\mathbb{B}}=\left\\{\chi_{0},\dots,\chi_{M}\right\\}$
sampled from $\tilde{\pi}_{\mathrm{e}}({\bm{a}}|{\bm{s}},t)$.

###  IV-D Reinforcement learning agents

sac [38] serves as a baseline for our experiments due to its sample efficiency
and good performance on previous experiments [4]. We include three variants of
imitation learning to investigate the benefit of expert knowledge for the RL
agent. First, we use reference state initialization to redefine the set of
initial states, to the set of states contained in the expert trajectories
${\mathbb{S}}_{0}=\left\\{{\bm{s}}\mid{\bm{s}}=\chi[k],\chi\in{\mathbb{B}},k\in\\{0,\dots,K-1\\}\right\\}$,
where we denote $\chi[k]$ as the $k$-th element of $\chi$. Starting the
episode from a state reached by the expert informs the agent about reachable
states and their reward in long-horizon tasks.

Secondly, we implement a state-based imitation reward, where the agent
receives an additional reward signal proportional to its closeness to an
expert trajectory in state space
$\tilde{r}({\bm{s}}_{k},{\bm{a}})=(1-\varsigma)r({\bm{s}}_{k},{\bm{a}})+\varsigma\operatorname{dist}({\bm{s}}_{k}-\chi[k])$,
where $0\leq\varsigma\ll 1$, and $\chi\in{\mathbb{B}}$. For the distance
function, we choose a re-scaled Gaussian function
$\operatorname{dist}({\bm{x}})=2^{-\kappa\|{\bm{x}}\|_{2}}$ with scaling
factor $\frac{1}{\kappa}$ as suggested in [6]. We further apply reference
state initialization when using the state-based imitation reward, as proposed
in [6].

Finally, we adapt the state-based imitation reward method to an action-based
imitation reward, where the agent receives an additional reward signal
proportional to the closeness of its action to the expert action
$\bar{r}({\bm{s}},{\bm{a}})=(1-\varsigma)r({\bm{s}},{\bm{a}})+\varsigma\operatorname{dist}({\bm{a}}-\bar{{\bm{a}}})$,
with $\bar{{\bm{a}}}\sim\tilde{\pi}_{\mathrm{e}}(\bar{{\bm{a}}}|{\bm{s}},t)$.
When using action-based imitation reward, we sample the expert policy
alongside the RL policy in every step but only execute the RL action.

##  V Experiments

This section presents the RL agents, shows the performance of the agents in
six out of eight benchmark environments, and discusses the results. Our
experiments aim to answer three main research questions:

  * •

Can RL be used to complete complex HRC tasks?

  * •

How beneficial is prior expert knowledge in solving these tasks?

  * •

Does the RL agent overfit to a limited amount of human animations in training?

(a)  (b)  (c)  (d)  (e)  (f)  (g)  (h)  (i)  (j)  (k)  (l)  (m)  Figure 3:
Evaluation performance during training of RL agents on human-robot gym. The
plots show the mean evaluation performance during training and the 95%
confidence interval in the mean metric obtained with bootstrapping when
training on five random seeds.

Reach Pick and Place Collaborative Lifting Robot-Human Handover Human-Robot
Handover Collaborative Stacking

Figure 4: Ablation study for overfitting to animation data in the training
process.

We present our results on six of the eight human-robot gym environments:
reach, pick and place, collaborative lifting, robot-human handover, human-
robot handover, and collaborative stacking. To evaluate the benefit of expert
knowledge in these complex tasks, we compare the four agents discussed in
Section IV-D with the expert. Fig. 3a shows our main results, where we
evaluate the performance every $2\text{$\cdot$}{10}^{5}$ training steps and
trained all agents on five random seeds. We report the success rate, which
indicates the rate at which the task was successful, and the reward normalized
to the range between the minimal possible reward and the average expert
reward. All plots show the mean evaluation performance during training and the
$95\text{\,}\%$ confidence interval (shaded area) in the mean metric
established with bootstrapping on $10^{4}$ resamples.

Our results show that the human-robot gym has a diverse set of tasks, from
which some are already solvable, e.g., reach and pick and place, some show
room for improvement, e.g., collaborative lifting and robot-human handover,
and some are not solvable with the investigated approaches, e.g., human-robot
handover and collaborative stacking. Comparing these results to the complexity
estimate in Table I, we infer that the two main factors for the difficulty of
a task are the complexity of the manipulation and the human dynamics. Handling
these two areas will be one of the main challenges for RL research in HRC.

The results in Fig. 3a further show that expert knowledge is beneficial in
environments with sparse rewards. In these environments, the action-based
imitation reward method shows higher or equal performance than the state-based
one. In the pick and place task, the action-based imitation reward approach
outperformed the expert policy and reached a nearly $100\text{\,}\%$ success
rate. Unfortunately, constructing the action-based imitation reward requires
an expert policy that can be queried online during training, which is not
given in many manipulation tasks. Interestingly, the agent trained with a
state-based imitation reward shows no significant improvement over the soft
actor-critic (SAC) agent trained only with reference state initialization in
our HRC environments. Our results indicate that starting the environment in
meaningful high-reward states significantly improves performance in sparse
reward settings. Future work could investigate if there are even more
effective forms of reference state initialization that require little to no
expert knowledge. Finally, expert knowledge does not improve performance in
our experiments with dense reward settings, such as reaching or collaborative
lifting. We assume this behavior stems from the fact that the additional
action-based and state-based imitation rewards resemble the dense environment
reward, yielding little additional information.

To address the concerns related to overfitting to the limited amount of human
animations in RL training, we conduct an ablation study on the collaborative
lifting task, which relies exceedingly on human animations. This study aims to
identify whether training an RL agent using a limited set of animations
instead of simulated behavior is satisfactory. Our dataset consists of nine
unique human animations, seven of which we use as training data, reserving the
remaining two for testing. We then perform a five-fold cross-evaluation, where
we select different training and testing animations on each split and train RL
agents on five random seeds per split. We report the average performance over
the splits and seeds and the $95\text{\,}\%$ confidence interval in the mean
metric of the trained SAC agent on the respective training animations (seen
data) and test animations (unseen data) in Fig. 4. The reward performance of
the trained agent on the unseen data is within the confidence interval of the
performance on the training data. Both mean reward and success rate are only
slightly lower on the unseen data, and the agent performs reasonably well.
Therefore, we conclude that overfitting to the human animation data is not a
significant problem of human-robot gym.

##  VI Conclusion

In conclusion, human-robot gym offers a practical benchmark for safe RL in
HRC, incorporating real-world animations pivotal for realistic HRC simulation.
Additionally, human-robot gym is the first benchmark for comparing safety
functions in HRC. Its unique provision of a pre-implemented safety shield
offers the opportunity to develop efficient HRC without designing a safety
function. Our evaluation insights reveal the importance of expert knowledge in
environments with sparse rewards, showing that an action-based imitation
reward is a promising approach if an expert is available online. In terms of
practical application, it is noteworthy that an agent trained in human-robot
gym was successfully deployed in actual HRC environments, as presented in our
prior work [31]. These tests underline the critical role human-robot gym will
play as an academic tool and as a practical approach for tangible robotic
issues.

## Acknowledgment

The authors gratefully acknowledge financial support by the Horizon 2020 EU
Framework Project CONCERT under grant 101016007.

## References

  * [1] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman _et al._ , “RT-1: Robotics Transformer for real-world control at scale,” _arXiv preprint arXiv:2212.06817_ , 2022\.
  * [2] R. Liu, F. Nageotte, P. Zanne, M. de Mathelin, and B. Dresp-Langley, “Deep reinforcement learning for the control of robotic manipulation: A focussed mini-review,” _Robotics_ , vol. 10, no. 1, pp. 1–13, 2021.
  * [3] F. Semeraro, A. Griffiths, and A. Cangelosi, “Human–robot collaboration and machine learning: A systematic review of recent research,” _Robotics and Computer-Integrated Manufacturing_ , vol. 79, 2023.
  * [4] J. Thumm and M. Althoff, “Provably safe deep reinforcement learning for robotic manipulation in human environments,” in _Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2022, pp. 6344–6350.
  * [5] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” _Journal of Machine Learning Research_ , vol. 22, no. 268, pp. 1–8, 2021\.
  * [6] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne, “DeepMimic: Example-guided deep reinforcement learning of physics-based character skills,” _ACM Transactions on Graphics_ , vol. 37, no. 4, pp. 1–14, 2018\.
  * [7] J. Ramírez, W. Yu, and A. Perrusquía, “Model-free reinforcement learning from expert demonstrations: A survey,” _Artificial Intelligence Review_ , vol. 55, no. 4, pp. 3213–3241, 2022.
  * [8] B. Zheng, S. Verma, J. Zhou, I. W. Tsang, and F. Chen, “Imitation learning: Progress, taxonomies and challenges,” _IEEE Transactions on Neural Networks and Learning Systems (Early Access)_ , pp. 1–16, 2022.
  * [9] E. Todorov, T. Erez, and Y. Tassa, “MuJoCo: A physics engine for model-based control,” in _Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2012, pp. 5026–5033.
  * [10] D. Vogt, S. Stepputtis, S. Grehl, B. Jung, and H. Ben Amor, “A system for learning continuous human-robot interactions from human-human demonstrations,” in _Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2017, pp. 2882–2889.
  * [11] A. Cunha, F. Ferreira, E. Sousa, L. Louro, P. Vicente, S. Monteiro, W. Erlhagen, and E. Bicho, “Towards collaborative robots as intelligent co-workers in human-robot joint tasks: What to do and who does it?” in _Proc. of the Int. Symp. on Robotics_ , 2020, pp. 1–8.
  * [12] D. Shukla, Ö. Erkent, and J. Piater, “Learning semantics of gestural instructions for human-robot collaboration,” _Frontiers in Neurorobotics_ , vol. 12, 2018.
  * [13] G. J. Maeda, G. Neumann, M. Ewerton, R. Lioutikov, O. Kroemer, and J. Peters, “Probabilistic movement primitives for coordination of multiple human–robot collaborative tasks,” _Autonomous Robots_ , vol. 41, no. 3, pp. 593–612, 2017.
  * [14] L. Roveda, J. Maskani, P. Franceschi, A. Abdi, F. Braghin, L. Molinari Tosatti, and N. Pedrocchi, “Model-based reinforcement learning variable impedance control for human-robot collaboration,” _Journal of Intelligent & Robotic Systems_, vol. 100, no. 2, pp. 417–433, 2020.
  * [15] Z. Deng, J. Mi, D. Han, R. Huang, X. Xiong, and J. Zhang, “Hierarchical robot learning for physical collaboration between humans and robots,” in _Proc. of the IEEE Int. Conf. on Robotics and Biomimetics (ROBIO)_ , 2017, pp. 750–755.
  * [16] S. Nikolaidis, R. Ramakrishnan, K. Gu, and J. Shah, “Efficient model learning from joint-action demonstrations for human-robot collaborative tasks,” in _Proc. of the ACM/IEEE Int. Conf. on Human-Robot Interaction (HRI)_ , 2015, pp. 189–196.
  * [17] Y. Zhu, J. Wong, A. Mandlekar, and R. Martín-Martín, “robosuite: A modular simulation framework and benchmark for robot learning,” _https://arxiv.org/abs/2009.12293_ , 2020.
  * [18] M. Mittal, C. Yu, Q. Yu, J. Liu, N. Rudin, D. Hoeller, J. L. Yuan, R. Singh, Y. Guo, H. Mazhar, A. Mandlekar, B. Babich, G. State, M. Hutter, and A. Garg, “Orbit: A unified simulation framework for interactive robot learning environments,” _IEEE Robotics and Automation Letters_ , vol. 8, no. 6, pp. 3740–3747, 2023.
  * [19] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, M. Anvari, M. Hwang, M. Sharma, A. Aydin, D. Bansal, S. Hunter, A. Lou, C. R. Matthews, I. Villa-Renteria, J. H. Tang, C. Tang, F. Xia, S. Savarese, H. Gweon, K. Liu, J. Wu, and L. Fei-Fei, “BEHAVIOR-1K: A benchmark for embodied AI with 1,000 everyday activities and realistic simulation,” in _Proc. of the Conf. on Robot Learning (CoRL)_ , vol. 205, 2022, pp. 80–93.
  * [20] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning,” in _Proc. of the Conf. on Robot Learning (CoRL)_ , 2020, pp. 1094–1100.
  * [21] P. Higgins, G. Y. Kebe, A. Berlier, K. Darvish, D. Engel, F. Ferraro, and C. Matuszek, “Towards making virtual human-robot interaction a reality,” in _Proc. of the Int. Workshop on Virtual, Augmented, and Mixed-Reality for Human-Robot Interactions (VAM-HRI)_ , 2021, pp. 1–5.
  * [22] T. Inamura and Y. Mizuchi, “SIGVerse: A cloud-based VR platform for research on multimodal human-robot interaction,” _Frontiers in Robotics and AI_ , vol. 8, pp. 1–19, 2021.
  * [23] Z. Erickson, V. Gangaram, A. Kapusta, C. K. Liu, and C. C. Kemp, “Assistive gym: A physics simulation framework for assistive robotics,” in _Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2020, pp. 10 169–10 176.
  * [24] R. Ye, W. Xu, H. Fu, R. K. Jenamani, V. Nguyen, C. Lu, K. Dimitropoulou, and T. Bhattacharjee, “RCareWorld: A human-centric simulation world for caregiving robots,” in _Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2022, pp. 33–40.
  * [25] Y.-W. Chao, C. Paxton, Y. Xiang, W. Yang, B. Sundaralingam, T. Chen, A. Murali, M. Cakmak, and D. Fox, “HandoverSim: A simulation framework and benchmark for human-to-robot object handovers,” in _Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2022, pp. 6941–6947.
  * [26] I. Uchendu, T. Xiao, Y. Lu, B. Zhu, M. Yan, J. Simon, M. Bennice, C. Fu, C. Ma, J. Jiao, S. Levine, and K. Hausman, “Jump-start reinforcement learning,” in _Proc. of the Int. Conf. on Machine Learning (ICML)_ , 2023, pp. 34 556–34 583.
  * [27] O. Yadan, “Hydra - a framework for elegantly configuring complex applications,” 2019.
  * [28] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight experience replay,” in _Proc. of the Int. Conf. on Neural Information Processing Systems (NeurIPS)_ , 2017, pp. 5055–5065.
  * [29] R. Li, A. Jabri, T. Darrell, and P. Agrawal, “Towards practical multi-object manipulation using relational reinforcement learning,” in _Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2020, pp. 4051–4058.
  * [30] J. Carpentier, G. Saurel, G. Buondonno, J. Mirabel, F. Lamiraux, O. Stasse, and N. Mansard, “The Pinocchio c++ library – a fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives,” in _Proc. of the Int. Symp. on System Integrations (SII)_ , 2019.
  * [31] J. Thumm, G. Pelat, and M. Althoff, “Reducing safety interventions in provably safe reinforcement learning,” 2023.
  * [32] ISO, “Robotics - safety requirements - part 1: Industrial robots,” International Organization for Standardization, Tech. Rep. DIN EN ISO 10218-1:2021-09 DC, 2021.
  * [33] D. Beckert, A. Pereira, and M. Althoff, “Online verification of multiple safety criteria for a robot trajectory,” in _Proc. of the IEEE Conf. on Decision and Control (CDC)_ , 2017, pp. 6454–6461.
  * [34] S. B. Liu and M. Althoff, “Online verification of impact-force-limiting control for physical human-robot interaction,” in _Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2021, pp. 777–783.
  * [35] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. MIT press, 2018.
  * [36] G. E. Uhlenbeck and L. S. Ornstein, “On the theory of the Brownian motion,” _Physical Review_ , vol. 36, no. 5, pp. 823–841, 1930.
  * [37] D. S. Lemons, _An Introduction to Stochastic Processes in Physics_. The Johns Hopkins University Press, 2002.
  * [38] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _Proc. of the Int. Conf. on Machine Learning (ICML)_ , 2018, pp. 1861–1870.

  *[RL]: reinforcement learning
  *[HRC]: human-robot collaboration
  *[SSM]: speed and separation monitoring
  *[PFL]: power and force limiting
  *[ISS]: invariably safe state
  *[PID]: proportional–integral–derivative
  *[SAC]: soft actor-critic