# NOMAD: A Natural, Occluded, Multi-scale Aerial Dataset, for Emergency
Response Scenarios

Arturo Miguel Russell Bernal, Walter Scheirer, Jane Cleland-Huang
Computer Science and Engineering Department, University of Notre Dame,
Indiana, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

With the increasing reliance on small Unmanned Aerial Systems (sUAS) for
Emergency Response Scenarios, such as Search and Rescue, the integration of
computer vision capabilities has become a key factor in mission success.
Nevertheless, computer vision performance for detecting humans severely
degrades when shifting from ground to aerial views. Several aerial datasets
have been created to mitigate this problem, however, none of them has
specifically addressed the issue of occlusion, a critical component in
Emergency Response Scenarios. Natural Occluded Multi-scale Aerial Dataset
(NOMAD) presents a benchmark for human detection under occluded aerial views,
with five different aerial distances and rich imagery variance. NOMAD is
composed of 100 different Actors, all performing sequences of walking, laying
and hiding. It includes 42,825 frames, extracted from 5.4k resolution videos,
and manually annotated with a bounding box and a label describing 10 different
visibility levels, categorized according to the percentage of the human body
visible inside the bounding box. This allows computer vision models to be
evaluated on their detection performance across different ranges of occlusion.
NOMAD is designed to improve the effectiveness of aerial search and rescue and
to enhance collaboration between sUAS and humans, by providing a new benchmark
dataset for human detection under occluded aerial views.

##  1 Introduction

Advances in technology, including improvements in edge computing and
Artificial Intelligence (AI), have led to increased use of small Unmanned
Aerial Systems (sUAS) across a broad range of applications [52, 69, 53], such
as emergency response [69, 53, 26, 6]. sUAS are empowered to perform Computer
Vision (CV) tasks, such as aerial surveillance and autonomous person detection
and tracking, where timely and efficient performance potentially can make the
difference between life or death [21, 27, 32]. Higher levels of sUAS autonomy,
supported by CV, increase collaboration between humans and sUAS, allowing
emergency responders to focus attention on mission level goals [1, 20] while
sUAS perform lower-level person detection tasks.

However, there are many open challenges in deploying CV on sUAS for emergency
response [19, 51]. These challenges include the non-trivial, highly prevalent
problem of occlusion, which occurs when targets of aerial search are often
partially hidden from view. For example, a drowning victim who is partially
submerged in water, people buried in debris following an earthquake, hidden by
smoke in a fire, or laying behind trees and rocks in search and rescue
missions. Occlusion could also be intentional when a suspect is hiding from
law-enforcement, caused by pose and image perspective, or introduced in far-
distance aerial views due to glare, shades, blur, and low resolution. Prior CV
research on occlusion has focused on generic object detection [47, 66], as
well as on pedestrian detection [59], demonstrating how occlusion drastically
affects model performance [59, 79, 82]. However, the occlusion problem is
exacerbated even further when shifting from ground to aerial views [58], where
additional challenges surrounding the incorporation of CV capable sUAS into
emergency response scenarios include the biased training datasets, coupled
with real-life challenges such as vibration, wind and atmospheric turbulence,
harsh weather and low visibility conditions, diverse scenery, and the need for
generalization at different distances and resolutions. CV systems deployed for
emergency response must be able to reliably handle person detection under all
of these conditions.

Figure 1: Development and characteristics of NOMAD. Integration of sUAS into
emergency response scenarios have aided first responders and rescued victims
[21, 27, 32, 31, 42, 15, 12, 36] (first column). Nevertheless, multiple
challenges inherent to these situations degrade CV performance and halt sUAS
full integration, including the highly prevalent problem of occlusion (second
column). We present NOMAD, Natural Occluded Multi-scale Aerial Dataset,
providing the research community with emergency response related videos and
selected frames, as well as rich metadata and annotations, including a
visibility label (third column). Facing emergency response scenarios, key
characteristics of our dataset are: Natural: diversity of filming locations,
cross-seasonal imagery, including winter scenarios, and a demographic
diversity on gender, age and race, ranging from 18 to 78 years old, and
including White Caucasians, Latinos, African descent, Asians, South Asians,
Middle Eastern and Pacific Islander; Occluded: 10 defined ranges of occlusion,
with a visibility label assigned to every bounding box; Multi-scale: five
different distances, ranging from 10m to 90m altitude, and a ground reference
view for every actor.

We therefore address these challenges through presenting NOMAD (Natural
Occluded Multi-scale Aerial Dataset), a benchmark dataset aimed at human
detection under occluded aerial views, as summarized in Fig. 1. NOMAD is
composed of 100 different actors, each performing sequences of walking, laying
and hiding. It includes 42,825 frames, extracted from 5.4k resolution videos.
Actors are manually annotated with a bounding box and a label describing 10
different visibility levels, categorized according to the percentage of the
human body visible inside the bounding box, allowing the detection performance
of CV models to be evaluated across 10 different ranges of occlusion. Figure 1
summarizes the key characteristics of our dataset, including: Natural:
representing a variety of natural and man-made locations; cross-seasonal
imagery, ranging from summer to winter scenarios; demographic variety on age
and race, ranging from 18 to 78 years old, and including White Caucasians,
Latinos, African descent, Asians, South Asians, Middle Eastern and Pacific
Islander; Occluded: with routines created to include occlusion and a
visibility label assigned to every bounding box annotated; Multi-scale: with
five different distances, ranging from 10m to 90m altitude, and a ground
reference view for every actor.

The remainder of this article is organized as follows. Section 2 presents
related work. Section 3 describes the data collection process. Section 4
describes the data curation, key-frame selection and data annotation. Section
5 discusses NOMAD characteristics and its potential uses. Section 6 reports
baseline results achieved using state-of-the-art CV detection models under
different levels of occlusion, and Sec. 7 summarizes the contributions of the
work.

##  2 Related Work

###  2.1 Mobile Robotics for Emergency Response

There are numerous challenges associated with integrating mobile robotics into
emergency response missions [19, 51, 69, 53, 70, 57]. Researchers, focusing on
ground mobile robots, have explored mapping of emergency scenes [60, 73, 64],
improved communication networks [49], and specialized architectures [33, 41].
User studies have demonstrated the benefits of including aerial robots in
emergency response [74], potentially working in collaboration with ground
robots [17], to enhance surveying and mapping capabilities [63]. Other studies
have explored the integration of additional sensors, such as ground
penetrating radar [62], or cellphone tracking for missing person search [3].
Finally, several researchers have explored efficient collaborations between
humans and sUAS at the intersection of software engineering and human computer
interaction [14, 1, 20, 2].

###  2.2 Real-World Object Detection

There are numerous challenges related to utilizing aerial CV for real-time
emergency response [47]. Real-time CV applications tend to leverage the latest
versions of the YOLO family [72, 39], as well as their modifications [56, 35,
55, 45], while other methods explore attention for object detection [54] and
multimodal techniques [4]. The most recent work has focused on incremental
learning of unknown classes, in the modality known as Open World Object
Detection [40, 50, 84, 80], as well as its variations [77, 81]. The challenges
of object detection under occlusion have also been studied [66, 59, 16].
Finally, techniques incorporating human perception have been explored for
object detection [61] and other machine learning tasks [67, 34, 37, 30, 24, 8,
9], demonstrating a plausible approach to handling occlusion [18, 65, 29, 82].

###  2.3 Aerial Datasets

While many datasets have been collected to aid aerial detection of humans in
search and rescue (SAR), none of them have addressed the critical issue of
occlusion. HERIDAL [10] comprises of approximately 500 labelled 4,000 by 3,000
pixel images suitable for object detection tasks. SARD comprises 1,981
manually labeled images extracted from video frames of persons simulating
search and rescue situations in roads, quarries, grass land, and forested
areas, under diverse weather conditions. However, both datasets lack rich
generalization characteristics and environmental diversity. The recently
published WiSARD [11] dataset, comprises the richest set of images associated
with wilderness SAR scenarios, with 33,786 labeled RGB images, 22,156 labeled
thermal images, and a subset consisting of 15,453 temporally synchronized
visual-thermal image pairs. In addition to the useful multimodal imagery, the
dataset includes environmental diversity across seasons and times of the day
and night. WiSARD represents the richest dataset for blind search in
wilderness scenarios, that is, search for any person on an area rather than
the search for an specific person; NOMAD provides richer demographic
diversity, includes man-made scenarios, provides rich metadata of actors,
controlled multi-scales, and provides a new benchmark for occlusion. It is the
only dataset, to our knowledge, to systematically address the issue of
occlusion.

The BIRDSAI, VisDrone and UAVDT [83, 7, 28], incorporate occlusion labels into
their annotations; however, they lack rich human metadata of humans. BIRDSAI
is a long-wave thermal infrared dataset containing nighttime images of animals
and humans in Southern Africa. While suitable for improving blind search of
persons in emergency scenarios, it only provides two levels of occlusion and
lacks person metadata. VisDrone consists of 288 video clips formed by 261,908
frames and 10,209 static images, and is captured by various drone-mounted
cameras, covering diverse locations, environments, objects (pedestrian,
vehicles, bicycles, etc.), and density. However, it provides only three levels
of occlusion and also lacks person metadata. Finally, while UAVDT provides
four levels of occlusion, it focuses purely on vehicles and not people.

BRIAR, MEVID, UAV-Human, P-DESTRE and PRAI-1581 [22, 25, 46, 43, 78] provide
rich metadata and are well suited for person re-identification. BRIAR and
MEVID datasets offer great diversity of camera views, with BRIAR providing
long range imagery of up to 1000m. BRIAR, so far, includes more than 350,000
still images and over 1,300 hours of video footage of approximately 1,000
subjects; MEVID, is part of the very-large-scale MEVA person activities
dataset [23] and comprises 158 unique people wearing 598 outfits collected
from 33 camera views. UAV-HUMAN includes 67,428 annotated video sequences of
119 subjects for action recognition, 22,476 annotated frames for pose
estimation, 41,290 annotated frames of 1,144 identities for person re-
identification, and 22,263 annotated frames for attribute recognition. While
these three datasets represent the most complete datasets for their given
purposes, none of them reference occlusion and all lack representation of
emergency response scenarios. Finally, PRAI-1581 provides 1,581 identities and
P-DESTRE provides rich metadata for 269 different identities, however, filming
distances are only up to 60m and 6.7m, respectively. Additional categorized
datasets can be found in [58].

Overall, NOMAD provides the demographic and environmental diversity needed to
tackle the person detection task of emergency response scenarios from aerial
views, while being the first dataset to include an occlusion metric for person
detection, and to provide detailed metadata and controlled multi-scale, making
it suitable for many other CV tasks as described in Sec. 5.

##  3 Data Collection Process

Our data collection process followed our IRB approved protocol 21-11-6913. In
a preliminary pilot study, our data collection procedure included strict
instructions regarding the percentage of the body that the actor should expose
to the sUAS’ camera at each step. However, we observed that these instructions
were difficult to follow causing disconnected movements, and so we replaced
the instructions with simpler ones that led to more natural behavior.

###  3.1 Recruitment

As per our IRB protocol, all participants were at least 18 years old. Further,
as the recruitment process evolved, participants from already well represented
demographic groups were excluded in order to achieve a balanced gender
distribution, a variety of age ranges, and a rich race distribution.

###  3.2 Location Selection

Approval for use of premises was attained from owners and responsible agencies
for all locations filmed in the dataset. The 12 locations included: 3
different Schools, 2 paintball courts, 1 forest park, 1 golf course, 1 lake
shore, 1 quarry, 2 farms, and 1 AMA flying field. This resulted in diversity
of locations, including both natural and man-made influenced, and provided a
variety of different types of obstacles for occlusion purposes.

###  3.3 Filming Sessions

All filming sessions followed IRB protocol guidelines with participants being
informed of the purpose of their performance, the activities to be completed,
and consent forms being signed. All flights were conducted by a certified FAA
Part 107 remote pilot, and all FAA protocols were followed, with air space
reserved through LAANC systems such as AirMap and DroneUp. Although efforts
were made to isolate the selected locations during the filming sessions,
unexpected persons appeared during a few of the filming sessions. In most
cases we stalled the filming until the person exited the scene; however, in a
few cases, these persons agreed to appear on the dataset, signing a consent
form. From here on, we call actors the participants performing the designated
routine, while non-actors are other participants who agreed to appear in the
dataset but were otherwise not engaged in the study.

Once the study introduction was completed, each actor was assigned a unique
obstacle at the filming location, and then given instructions for performing
the standard routine with respect to their obstacle as follows:

  * •

Starting Frame: With few exceptions, the first frame represented a view of the
actor completely visible.

  * •

Hiding: All actors were instructed to hide behind their obstacle two times,
with small variations in their hiding trajectory. This step allowed us to
obtain varying degrees of occluded aerial views.

  * •

Laying: To provide a variety of poses, actors were asked to lay down when
completely visible and when partially occluded by their obstacle.

  * •

Walking: Finally, actors performed a small walking trajectory at the end of
their routine.

  * •

General instructions: actors were informed of the dataset’s focus on emergency
response scenarios, and were therefore asked to position themselves as if they
were hiding, trying to be rescued, or in need of help.

For the water routines, where the primary occlusion source was the water
itself, small but important variations in instructions were given to simulate
various drowning scenarios. All actors were asked to repeat their routine five
times, with the sUAS set at five progressively distant locations set to 10m,
30m, 50m, 70m, and 90m, with the distance measured, through the sUAS’ feedback
from the First Person View (FPV) screen, horizontally and vertically from the
expected starting point of the actor. Figure 2 illustrates the sUAS position
at a distance of 10m.

Additionally, a reference view of the actor was filmed, with the sUAS
positioned a few meters in front of the actor, while the actor performed
360°rotations. The first rotation involved arms hanging down and the second
with arms extended up, providing multiple views of the actor at ground level.
Finally, true negatives were also filmed by asking the actor to locate
himself/herself outside of the camera view; please note that in a few cases
true negatives may still contain consented non-actor participants. At the
conclusion of the session, actors were given a 20$USD prepaid card.

Figure 2: Filming process. Sample positioning of the sUAS at 10m horizontally
and vertically from the actor’s starting location.

##  4 Data Annotation Process

###  4.1 Data curation

Although efforts were made to avoid filming non-participants, during the
revision of the films, unexpected persons were observed on a couple of videos.
For videos where the non-participant was only visible at the beginning, at the
end, or at non-keyframes, trimming the video was a direct solution,
representing no impact to the quality of the data. Nevertheless, situations
where found where trimming the portion of the video where the non-participant
appeared on screen would represent a loss of information of the actor’s
performance; these situations were solved by blacking out the non-participant
area of the frames.

###  4.2 Metadata

Metadata provided in this dataset can be divided into Demographic and
Environmental categories. It includes all descriptive information about the
actor, including outfit descriptions that could aid in re-identification
tasks. Full list of metadata can be found in Tab. S1 in the Supplemental
Material. Insights into selecting metadata factors was obtained through a
previous series of semi-structured interviews with emergency responders, under
IRB protocol 19-04-5269, to determine search terms used for describing missing
persons. Clothing descriptions may include up to five words for salient
figures. Hair length uses the same metric for males and females, with bald
meaning absence of noticeable hair, short meaning ear-length, medium ranging
from ear- to shoulder-length, and long meaning longer than shoulder-length.
The Location descriptor School (Nature) aggregates filming sessions where the
researcher should expect nature domination despite it being filmed at a school
premises. The reported weather information was obtained from the nearest
weather station to the filming location. Finally, the Exposure Value (EV) was
separated from the Video descriptor. While the Video descriptor is a constant
for all actors, the EV parameter was found to be different than 0 on a couple
of films, indicating a change in the illumination, which is a relevant
parameter for computer vision tasks [75]. Table S2 from the Supplemental
Material displays the predefined lists of available colors for describing
clothing and hair color. The ranges were selected to include the most common
colors across the hue range on the HSV color space; colors for the Hair
descriptor were selected based on emergency responders’ classifiers.

###  4.3 Keyframe selection

Manually labeling every frame from the films would have been infeasible, and
therefore we selected 85 keyframes at each of the five distances for every
actor. This resulted in 425 keyframes per actor, with the exception of
Actor001, for whom 750 keyframes were selected. Keyframe selection was
performed in accordance to the following guidelines: (1) the 85 selected
frames tracked the actor across their entire routine, (2) each starting and
ending frame of a trajectory was selected as a keyframe, (3) each change in
direction of the actor’s trajectory generated a new keyframe, (4) the set of
keyframes included different poses (e.g., standing, sitting, laying down), and
finally (5) all keyframes had at least a part of the actor visible.

Finally, when the actor interacts with an obstacle, a custom sampling is
performed to obtain views with different levels of occlusion as the actor
moves behind and away from an obstacle. Figure 3(a) illustrates a sample
routine with 12 keypoints selected following these guidelines. The red-dashed
arrows indicate where sampling for occluded views would be performed. Activity
labels were added to each keyframe as: Walking, Laying, Hiding, Hiding
(Laying), Swimming, Drowning. Figure 3(b) shows sample labels for the 12
keypoints illustrated on Fig. 3(a).

###  4.4 Annotations

All 42,825 selected frames were sent to Labelbox [44], a labelling company who
employed expert annotators to add bounding boxes and visibility labels to all
images.

####  4.4.1 Occlusion Label

Figure 4 displays how percentages were assigned to each body part of a person.
The following is the procedure used to calculate the visibility label, that
is, the amount of an actor that was visible at a particular instant: (1) Given
an image, identify the body parts of a person that are visible. (2) Review the
percentages of the identified body parts based on Fig. 4. (3) If less than
half of a body part is visible, assign half of the percentage indicated in
Fig. 4. (4) If more than half of the body part is visible, assign the full
percentage as indicated in Fig. 4. (5) Add up the percentages obtained from
each body part. (6) Assign the sum to one of the ten ranges of visibility,
with upper bounds of 10 to 100. For example, selecting 10 means that the sum
obtained from the percentages is greater than zero but less than or equal to
10, while selecting 20 means that the sum is greater than 10 and less than or
equal to 20, and so on.

Please note that shadows were not considered to be part of the human, and
under normal circumstances, the actor’s own clothing are not treated as a
source of occlusion. Also note that although we are reporting a visibility
metric, this is just the inverse of the occluded amount of the actor’s body.

(a)

Keyframes | Activity
---|---
1 - 2 | Walking
2 - 3 | Hiding
3 - 5 | Walking
5 - 6 | Hiding
6 - 7 | Walking
7 | Laying
7 - 8 | Walking
8 - 10 | Hiding (L)
10 - 12 | Walking
(b)

Figure 3: Keyframe selection process. (a) Sample routine with 12 keyframes
selected. Sampling for occluded views is indicated by red-dashed arrows. (b)
Sample Activity labels for the keyframes illustrated. Hiding (L) represents
Hiding (Laying). Figure 4: Visibility label calculation. Percentages assigned
to each body part of a person.

##  5 Dataset Characteristics

NOMAD provides 500 videos of 100 different actors, with each actor performing
routines at five different distances, set as 10m, 30m, 50m, 70m, and 90m
horizontally and vertically from the actor’s initial position. It also
provides 500 true negative videos of a couple of seconds duration, with each
true negative video corresponding to one routine video. Finally, one reference
video per actor is provided. Videos’ duration ranges from 30s to 180s
according to the actor’s pace. This resulted in 42,825 frames manually
annotated with a bounding box and visibility label. All videos are of 30fps,
MP4-H265 coding, 5.4k video quality, with all frames being of 5472 by 3078
pixels.

###  5.1 Natural

Figure 5 shows the distribution of the 100 actors with respect to their
filming locations. The variety of locations provides coverage of natural and
man-made environments, and is aimed at training CV models for effectively
supporting a wide range of emergency response scenarios.

To further increase robustness of our dataset in terms of environmental
conditions, cross-seasonal imagery was collected, with temperatures ranging
from 30F to 90F, wind speeds of 0MPH to 20MPH, morning, afternoon, and evening
sessions, capturing hot sunny summer days, autumn colorful scenes, and
winter’s snowy conditions. Finally, we made every effort to mitigate potential
demographic bias to support fair and equitable emergency response. Figure 6
presents the distributions of gender, age and race. The gender distribution
shows a 50/50 male/female distribution, and although age distribution shows
that the majority of the population was younger than 30 years old, actors
across the range of 30 to 78 years old are still present in significant
percentage. Finally, we show a comparison between our race distribution and
the USA race distribution [13, 38], showing an improvement with the purpose of
generalizing CV models and mitigating potential biases [71, 48]. While the USA
federal census does not consider Latino/Hispanic as a race, and distribute
their classification as an ethnicity distributed across races [13], we have
incorporated it as a race in alignment with its recognition as a separate
class by current computer vision models [68]. Lastly, although the non-rigid
aspect of our routine creates uncertainty about specific levels of our
visibility label, it allows the actor to provide data using more natural
behaviour, adding fidelity to the actor’s performance.

Figure 5: Distribution of the filming locations for the 100 actors.

###  5.2 Occluded

NOMAD provides the data needed to face occluded persons’ detection during
high-pressure, life-or-death emergencies. It labels each bounding box with the
degree of visibility on 10 levels, providing a representative number of frames
at each level as shown in Fig. 7. The higher amount of frames at lower
visibilities responds to the manual selection process as well as to the
increasing annotation difficulty and additional sources of occlusion at
further distances.

###  5.3 Multi-scale

Effective collaboration between sUAS and humans aims to exploit each of their
individual strengths. sUAS have the ability to quickly scan large areas from
greater altitudes, or to provide a focused close-up view of the target. NOMAD
provides five different aerial distances, supporting both generalized models
or models specialized for each distance. Table S3 from the Supplemental
Material shows the expected maximum Ground Sampling Distance (GSD) for the
five different distances, assuming that the actor is on the camera optical
axis [5]. This is not always true as the actors moved to perform their routine
and the camera gimbal position was often set to avoid potential areas of non-
participants or areas outside the filming premises. This moved the actors away
from the camera optical axis, increasing their GSD, and decreasing the number
of pixels representing them, as well as creating a non-fixed pitch and adding
real variance.

(a)

(b)

(c)

(d)

Figure 6: Distribution of the demographic descriptors of (a) Gender, (b) Age,
and (c) Race, for our 100 actors. Our race distribution is compared to the (d)
USA race distribution, improving generalization by mitigating possible biases.
Figure 7: Distribution of the visibility label across the 42,825 manually
annotated frames.

###  5.4 Computer Vision Uses

The characteristics of NOMAD provide an environment to improve emergency
response in four main areas of CV:

  * •

Occlusion benchmark: the effort of NOMAD’s ten levels of visibility aims to
provide a new benchmark dataset to assess the research community’s
improvements on person detection under occlusion, a previously under-explored
factor in aerial datasets.

  * •

Person detection: search and rescue scenarios in remote areas tend to search
for any person (i.e., blind search). The demographic and environmental
diversity provided by NOMAD, as well as its multi-scale component, supported
by the bounding boxes’ annotations, can be leveraged to improve this general
CV task.

  * •

Person re-identification: additional to blind search, descriptions of the
searched person translate the detection task to a CV re-identification
problem, especially on crowded scenarios. NOMAD provides a rich metadata and a
reference view of every actor to support re-identification tasks from aerial
views.

  * •

Person tracking: in many emergency response scenarios, the aim is to detect
and then track. Due to the strategic selection of manually labelled keyframes,
NOMAD allows the assessment of tracking techniques, following the actor’s key
movements and changes of direction throughout their full routine.

##  6 Computer Vision Models metrics

To demonstrate the use of NOMAD for benchmarking CV models at varying levels
of occlusion, we compared the performance of three state-of-the-art CV models.
Our first model was YOLOv8 from Ultralytics [39], representing the most recent
upgrade to the YOLO family. YOLOv8 supports real-time detection with limited
computational and memory resources, matching the requirements for sUAS-based
aerial detection. Additionally, we selected a FasterRCNN and a RetinaNet model
from the Detectron2 library [76]. The specific versions tested are YOLOv8l,
FasterRCNN-R101-FPN, and RetinaNet-R101-FPN, with a reported mAP@0.5:0.95 of
52.9, 42.0 and 40.4 on the COCO benchmark, respectively. Both libraries
provide a higher mAP model for YOLOv8 (YOLOv8x) and FasterRCNN
(FasterRCNN-X101-FPN), nevertheless, the latency of these other models
increases substantially compared to the gained mAP.

(a)

(b)

(c)

Figure 8: Performance across different levels of occlusion of (a) YOLOv8l’s,
(b) FasterRCNN-R101’s, (c) RetinaNet-R101’s pretrained weights when tested on
NOMAD, with the task of person detection. Oclussion increases as the level of
visibility decreases, therefore, mAP scores fall drastically as we increase in
distance and occlusion. The higher performance of the models at the closest
distance is expected as the data resembles ground view from COCO training
data, nevertheless, mAP scores fall significantly as we increase in occlusion
even for the closest distance, calling for improvement and robustness of
models against occlusion in aerial views.

For evaluation purposes, 10 folds of 10 actors each were randomly created,
with a constant seed for reproducibility across models. From each fold, 50
tests were performed, one per each distance-visibility (5 distances, 10
visibilities), with the results of these 50 tests being averaged across the 10
folds. Figure 8 shows the averaged mAP@0.5:0.95 score and standard deviation
of the CV models against different levels of occlusion and distances. We can
observe that YOLOv8l performs better on the closest distance than the other
models, a result supported by the higher initial mAP reported; nevertheless
all models suffer from critical degradation as the distance increases. This
behaviour is expected as their training data is focused on ground views rather
than aerial ones. Finally, although this degradation is expected to be
mitigated by fine-tuning the models with aerial data, the results emphasize
the degradation problem that occlusion represents, for even though the models
achieve decent scores at the closest distance with full visibility, the mAP
values drastically drop as the occlusion increases. The usefulness of NOMAD to
the research community can be justified from the previous baseline by three
reasons: (1) NOMAD is built with a real-world variance imagery (Natural),
making it a fair benchmark towards emergency response scenarios; (2) occlusion
on person detection can be assessed thanks to the granularity of our
visibility label; (3) the multi-scale characteristic allows occlusion to be
assessed also across different distances, key to the improvement of aerial
detection on sUAS. To exemplify the difficulty of person detection on
emergency response scenarios, Fig. 9 shows imagery from our tests with
increasing difficulty, due to distance and occlusion.

(a)

(b)

(c)

(d)

Figure 9: Test samples. (a) Easy sample at 10m and 100 visibility. (b) Medium
level difficulty sample (50m, 100 visibility). (c) Hard sample due to heavy
occlusion (10m, 30 visibility). (d) Hard sample due to high distance and light
occlusion (90m, 90 visibility).

##  7 Future Work and Conclusions

The structure and characteristics of NOMAD offer many opportunities for
improvements in aerial human detection, recognition and tracking, especially
the following:

  * •

Detection under occlusion: NOMAD allows us to explore and understand the
limits of detection under occluded views, with future work focusing on
improving CV models’ performance by exploiting human psychophysical metrics
and temporal information under these views.

  * •

Person re-identification: Addressing emergency response scenarios, future work
will focus on improving person re-identification through leveraging software
architectures that support hybrid onboard/offboard solutions and integrate the
human into the loop.

  * •

Real-world deployment: We have found through our own experiences in deploying
CV on sUAS that there are major additional challenges and some degradation in
results. Using the models trained on NOMAD, we will deploy and evaluate
occlusion-ready CV models on physical sUAS.

In conclusion, as indicated by the results of our baseline evaluation,
occlusion represents a non-trivial challenge that remains to be tackled.
NOMAD’s characteristics of Natural, Occluded, Multi-scale aerial views,
provide a new benchmark dataset for tackling this challenge, and can serve as
the next step in improving the accuracy of aerial search-and-detection for
emergency response.

## References

  * [1] Ankit Agrawal, Sophia J. Abraham, Benjamin Burger, Chichi Christine, Luke Fraser, John M. Hoeksema, Sarah Hwang, Elizabeth Travnik, Shreya Kumar, Walter J. Scheirer, Jane Cleland-Huang, Michael Vierhauser, Ryan Bauer, and Steve Cox.  The next generation of human-drone partnerships: Co-designing an emergency response system.  In Regina Bernhaupt, Florian ’Floyd’ Mueller, David Verweij, Josh Andres, Joanna McGrenere, Andy Cockburn, Ignacio Avellino, Alix Goguey, Pernille Bjøn, Shengdong Zhao, Briane Paul Samson, and Rafal Kocielnik, editors, CHI ’20: CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, April 25-30, 2020, pages 1–13. ACM, 2020.
  * [2] Md Nafee Al Islam, Muhammed Tawfiq Chowdhury, Ankit Agrawal, Michael Murphy, Raj Mehta, Daria Kudriavtseva, Jane Cleland-Huang, Michael Vierhauser, and Marsha Chechik.  Configuring mission-specific behavior in a product line of collaborating small unmanned aerial systems.  Journal of Systems and Software, 197:111543, 2023.
  * [3] Antonio Albanese, Vincenzo Sciancalepore, and Xavier Costa-Pérez.  Sardo: An automated search-and-rescue drone-based solution for victims localization.  IEEE Transactions on Mobile Computing, 21(9):3312–3325, 2022.
  * [4] Kirsnaragavan Arudpiragasam, Taraka Rama Krishna Kanth Kannuri, Klaus Schwarz, Michael Hartmann, and Reiner Creutzburg.  Real-time pedestrian detection using radar-camera fusion and clustering.  In Multimodal Image Exploitation and Learning 2023, volume 12526, pages 89–103. SPIE, 2023.
  * [5] Navaneeth Balamuralidhar, Sofia Tilon, and Francesco Nex.  Multeye: Monitoring system for real-time vehicle detection, tracking and speed estimation from uav imagery on edge-computing platforms.  Remote Sensing, 13(4), 2021.
  * [6] Arturo Miguel Russell Bernal and Jane Cleland-Huang.  Hierarchically organized computer vision in support of multi-faceted search for missing persons.  In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), pages 1–7. IEEE, 2023.
  * [7] Elizabeth Bondi, Raghav Jain, Palash Aggrawal, Saket Anand, Robert Hannaford, Ashish Kapoor, Jim Piavis, Shital Shah, Lucas Joppa, Bistra Dilkina, et al.  Birdsai: A dataset for detection and tracking in aerial thermal infrared videos.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1747–1756, 2020.
  * [8] Aidan Boyd, Daniel Moreira, Andrey Kuehlkamp, Kevin Bowyer, and Adam Czajka.  Human saliency-driven patch-based matching for interpretable post-mortem iris recognition.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pages 701–710, January 2023.
  * [9] Aidan Boyd, Patrick Tinsley, Kevin W Bowyer, and Adam Czajka.  Cyborg: Blending human saliency into the loss improves deep learning-based synthetic face detection.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6108–6117, 2023.
  * [10] Dunja Božić-Štulić, Željko Marušić, and Sven Gotovac.  Deep learning approach in aerial imagery for supporting land search and rescue missions.  International Journal of Computer Vision, 127(9):1256–1278, 2019.
  * [11] Daniel Broyles, Christopher R. Hayner, and Karen Leung.  Wisard: A labeled visual and thermal image dataset for wilderness search and rescue.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9467–9474, 2022.
  * [12] Brunswick County Sheriff’s Office, NC.  Teamwork makes the dream work, 2022.  https://www.facebook.com/brunswicksheriffNC/posts/355462306611443, Last accessed on 2023-08-27.
  * [13] Census Bureau.  US census bureau quick facts, 2023.  https://www.census.gov/quickfacts/, Last accessed on 2023-08-06.
  * [14] J. Casper and R.R. Murphy.  Human-robot interactions during the robot-assisted urban search and rescue response at the world trade center.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 33(3):367–385, 2003.
  * [15] CBS NEWS.  Italy police use drone to catch suspected arsonist in the act as wildfires char calabria, 2023.  https://www.cbsnews.com/news/italy-fire-drone-arson-suspect-arrest-calabria-wildfires-europe-algeria/, Last accessed on 2023-08-27.
  * [16] Himanshu Chandel and Sonia Vatta.  Occlusion detection and handling: a review.  International Journal of Computer Applications, 120(10), 2015.
  * [17] Dimitrios Chatziparaschis, Michail G Lagoudakis, and Panagiotis Partsinevelos.  Aerial and ground robot collaboration for autonomous mapping in search and rescue missions.  Drones, 4(4):79, 2020.
  * [18] Xiaowu Chen, Qing Li, Dongyue Zhao, and Qinping Zhao.  Occlusion cues for image scene layering.  Computer Vision and Image Understanding, 117(1):42–55, 2013.
  * [19] Muhammed Tawfiq Chowdhury and Jane Cleland-Huang.  Engineering challenges for ai-supported computer vision in small uncrewed aerial systems.  In 2023 IEEE/ACM 2nd International Conference on AI Engineering – Software Engineering for AI (CAIN), pages 158–170, 2023.
  * [20] Jane Cleland-Huang, Ankit Agrawal, Michael Vierhauser, Michael Murphy, and Mike Prieto.  Extending MAPE-K to support human-machine teaming.  In Bradley R. Schmerl, Martina Maggio, and Javier Cámara, editors, International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS 2022, Pittsburgh, PA, USA, May 22-24, 2022, pages 120–131. ACM/IEEE, 2022.
  * [21] CNN.  Drone lifeguard saves 14-year-old from drowning, 2022.  https://www.cnn.com/videos/us/2022/07/25/drone-rescues-teenager-drowning-spain-orig-cb-aw.cnn, Last accessed on 2023-08-27.
  * [22] David Cornett, Joel Brogan, Nell Barber, Deniz Aykac, Seth Baird, Nicholas Burchfield, Carl Dukes, Andrew Duncan, Regina Ferrell, Jim Goddard, et al.  Expanding accurate person recognition to new altitudes and ranges: The briar dataset.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 593–602, 2023.
  * [23] Kellie Corona, Katie Osterdahl, Roderic Collins, and Anthony Hoogs.  Meva: A large-scale multiview, multimodal video dataset for activity detection.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1060–1068, 2021.
  * [24] Adam Czajka, Daniel Moreira, Kevin Bowyer, and Patrick Flynn.  Domain-specific human-inspired binarized statistical image features for iris recognition.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 959–967, 2019.
  * [25] Daniel Davila, Dawei Du, Bryon Lewis, Christopher Funk, Joseph Van Pelt, Roderic Collins, Kellie Corona, Matt Brown, Scott McCloskey, Anthony Hoogs, et al.  Mevid: Multi-view extended videos with identities for video person re-identification.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1634–1643, 2023.
  * [26] DJI.  Drone rescues around the world, 2023.  https://enterprise.dji.com/drone-rescue-map/, Last accessed on 2023-08-04.
  * [27] Drones.R.Africa.  Drone helps save hiker on table mountain, cape town, 2019.  https://dronenews.africa/drone-helps-save-man-on-table-mountain-cape-town/, Last accessed on 2023-08-27.
  * [28] Dawei Du, Yuankai Qi, Hongyang Yu, Yifan Yang, Kaiwen Duan, Guorong Li, Weigang Zhang, Qingming Huang, and Qi Tian.  The unmanned aerial vehicle benchmark: Object detection and tracking.  In Proceedings of the European conference on computer vision (ECCV), pages 370–386, 2018.
  * [29] Justin Dulay, Sonia Poltoratski, Till S Hartmann, Samuel E Anthony, and Walter J Scheirer.  Guiding machine perception with psychophysics.  arXiv preprint arXiv:2207.02241, 2022.
  * [30] Justin Dulay and Walter J Scheirer.  Using human perception to regularize transfer learning.  arXiv preprint arXiv:2211.07885, 2022.
  * [31] EL PAÍS.  Gallery aerial images reveal scale of earthquake devastation in turkey and syria, 2023.  https://english.elpais.com/international/2023-02-09/gallery-aerial-images-reveal-scale-of-earthquake-devastation-in-turkey-and-syria.html, Last accessed on 2023-08-27.
  * [32] EMS1.  Drone helps N.C. first responders rescue missing teen, 2023.  https://www.ems1.com/drones/articles/drone-helps-nc-first-responders-rescue-missing-teen-igX40AfVSNKreUJk/, Last accessed on 2023-08-27.
  * [33] Han Fan, Victor Hernandez Bennetts, Erik Schaffernicht, and Achim J Lilienthal.  Towards gas discrimination and mapping in emergency response scenarios using a mobile robot with an electronic nose.  Sensors, 19(3):685, 2019.
  * [34] Ruth C Fong, Walter J Scheirer, and David D Cox.  Using human brain activity to guide machine learning.  Scientific reports, 8(1):5397, 2018.
  * [35] Prakhar Ganesh, Yao Chen, Yin Yang, Deming Chen, and Marianne Winslett.  Yolo-ret: Towards high accuracy real-time object detection on edge gpus.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3267–3277, January 2022.
  * [36] Global News.  Police drone locates missing 81-year-old woman in north carolina, 2017.  https://globalnews.ca/news/3851197/police-drone-locates-missing-81-year-old-woman-in-north-carolina/, Last accessed on 2023-08-27.
  * [37] Samuel Grieggs, Bingyu Shen, Greta Rauch, Pei Li, Jiaqi Ma, David Chiang, Brian Price, and Walter J. Scheirer.  Measuring human perception to improve handwritten document transcription.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6594–6601, 2022.
  * [38] Connie Hanzhang Jin, Ruth Talbot, and Hansi Lo Wang.  What the new census data shows about race depends on how you look at it, 2021.  https://www.npr.org/2021/08/13/1014710483/2020-census-data-us-race-ethnicity-diversity, Last accessed on 2023-08-06.
  * [39] Glenn Jocher, Ayush Chaurasia, and Jing Qiu.  Ultralytics yolov8, 2023.
  * [40] K J Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian.  Towards open world object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5830–5840, June 2021.
  * [41] Tetsushi Kamegawa, Taichi Akiyama, Satoshi Sakai, Kento Fujii, Kazushi Une, Eitou Ou, Yuto Matsumura, Toru Kishutani, Eiji Nose, Yusuke Yoshizaki, et al.  Development of a separable search-and-rescue robot composed of a mobile robot and a snake robot.  Advanced Robotics, 34(2):132–139, 2020.
  * [42] KPRC 2 Click2Houston.  Deputies use drone to find 3-year-old missing in washington brush, 2023.  https://www.youtube.com/watch?v=tlwG-f_lRnM, Last accessed on 2023-08-27.
  * [43] SV Aruna Kumar, Ehsan Yaghoubi, Abhijit Das, BS Harish, and Hugo Proença.  The p-destre: A fully annotated dataset for pedestrian detection, tracking, and short/long-term re-identification from aerial devices.  IEEE Transactions on Information Forensics and Security, 16:1696–1708, 2020.
  * [44] Labelbox.  Labelbox, 2023.  [Online] Available: https://labelbox.com.
  * [45] Jeonghun Lee and Kwang-il Hwang.  Yolo with adaptive frame control for real-time object detection applications.  Multimedia Tools and Applications, 81(25):36375–36396, 2022.
  * [46] Tianjiao Li, Jun Liu, Wei Zhang, Yun Ni, Wenqian Wang, and Zhiheng Li.  Uav-human: A large benchmark for human behavior understanding with unmanned aerial vehicles.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16266–16275, 2021.
  * [47] Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen.  Deep learning for generic object detection: A survey.  International journal of computer vision, 128:261–318, 2020.
  * [48] Steve Lohr.  Facial recognition is accurate, if you’re a white guy, 2018.  https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html, Last accessed on 2023-08-06.
  * [49] Georgios Loukas, Stelios Timotheou, and Erol Gelenbe.  Robotic wireless network connection of civilians for emergency response operations.  In 2008 23rd International Symposium on Computer and Information Sciences, pages 1–6, 2008.
  * [50] Yuqing Ma, Hainan Li, Zhange Zhang, Jinyang Guo, Shanghang Zhang, Ruihao Gong, and Xianglong Liu.  Annealing-based label-transfer learning for open world object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11454–11463, June 2023.
  * [51] Thomas Manzini and Robin Murphy.  Open problems in computer vision for wilderness sar and the search for patricia wu-murad.  arXiv preprint arXiv:2307.14527, 2023.
  * [52] MarketsAndMarkets.  UAV market by point of sale, systems, platform, function, end use, application, type, mode of operation, mtow, range and region - global forecast to 2027, 2022.  https://www.marketsandmarkets.com/Market-Reports/unmanned-aerial-vehicles-uav-market-662.html, Last accessed on 2023-08-04.
  * [53] Patrick McEnroe, Shen Wang, and Madhusanka Liyanage.  A survey on the convergence of edge computing and AI for UAVs: Opportunities and challenges.  IEEE Internet of Things Journal, 9(17):15435–15459, 2022.
  * [54] Shuyu Miao, Shanshan Du, Rui Feng, Yuejie Zhang, Huayu Li, Tianbi Liu, Lin Zheng, and Weiguo Fan.  Balanced single-shot object detection using cross-context attention-guided network.  Pattern recognition, 122:108258, 2022.
  * [55] Chintakindi Balaram Murthy, Mohammad Farukh Hashmi, and Avinash G Keskar.  Efficientlitedet: a real-time pedestrian and vehicle detection algorithm.  Machine Vision and Applications, 33(3):47, 2022.
  * [56] Jamuna S Murthy, GM Siddesh, Wen-Cheng Lai, BD Parameshachari, Sujata N Patil, and KL Hemalatha.  Objectdetect: A real-time object detection framework for advanced driver assistant systems using yolov5.  Wireless Communications and Mobile Computing, 2022, 2022.
  * [57] Keiji Nagatani, Seiga Kiribayashi, Yoshito Okada, Kazuki Otake, Kazuya Yoshida, Satoshi Tadokoro, Takeshi Nishimura, Tomoaki Yoshida, Eiji Koyanagi, Mineo Fukushima, et al.  Emergency response to the nuclear accident at the fukushima daiichi nuclear power plants using mobile rescue robots.  Journal of Field Robotics, 30(1):44–63, 2013.
  * [58] Kien Nguyen, Clinton Fookes, Sridha Sridharan, Yingli Tian, Feng Liu, Xiaoming Liu, and Arun Ross.  The state of aerial surveillance: A survey.  arXiv preprint arXiv:2201.03080, 2022.
  * [59] Chen Ning, Li Menglu, Yuan Hao, Su Xueping, and Li Yunhong.  Survey of pedestrian detection with occlusion.  Complex & Intelligent Systems, 7:577–587, 2021.
  * [60] Farzad Niroui, Kaicheng Zhang, Zendai Kashino, and Goldie Nejat.  Deep reinforcement learning robot for search and rescue applications: Exploration in unknown cluttered environments.  IEEE Robotics and Automation Letters, 4(2):610–617, 2019.
  * [61] Chen Pan and Wei Qi Yan.  Object detection based on saturation of visual perception.  Multimedia Tools and Applications, 79:19925–19944, 2020.
  * [62] Maria Gaia Pensieri, Mauro Garau, and Pier Matteo Barone.  Drones as an integral part of remote sensing technologies to help missing people.  Drones, 4(2):15, 2020.
  * [63] Aveek Purohit, Zheng Sun, Frank Mokaya, and Pei Zhang.  Sensorfly: Controlled-mobile sensing platform for indoor emergency response applications.  In Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks, pages 223–234, 2011.
  * [64] Frédéric Py, Giulia Robbiani, Giancarlo Marafioti, Yu Ozawa, Masahiro Watanabe, Kenichi Takahashi, and Satoshi Tadokoro.  Smurf software architecture for low power mobile robots: experience in search and rescue operations.  In 2022 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pages 264–269, 2022.
  * [65] Brandon RichardWebster, Samuel E. Anthony, and Walter J. Scheirer.  Psyphy: A psychophysics driven evaluation framework for visual recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9):2280–2286, 2019.
  * [66] Kaziwa Saleh, Sándor Szénási, and Zoltán Vámossy.  Occlusion handling in generic object detection: A review.  In 2021 IEEE 19th World Symposium on Applied Machine Intelligence and Informatics (SAMI), pages 000477–000484, 2021.
  * [67] Walter J. Scheirer, Samuel E. Anthony, Ken Nakayama, and David D. Cox.  Perceptual annotation: Measuring human vision to improve computer vision.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(8):1679–1686, 2014.
  * [68] Sefik Ilkin Serengil and Alper Ozpinar.  Hyperextended lightface: A facial attribute analysis framework.  In 2021 International Conference on Engineering and Emerging Technologies (ICEET), pages 1–4. IEEE, 2021.
  * [69] Hazim Shakhatreh, Ahmad H. Sawalmeh, Ala Al-Fuqaha, Zuochao Dou, Eyad Almaita, Issa Khalil, Noor Shamsiah Othman, Abdallah Khreishah, and Mohsen Guizani.  Unmanned aerial vehicles (uavs): A survey on civil applications and key research challenges.  IEEE Access, 7:48572–48634, 2019.
  * [70] Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, et al.  The limits and potentials of deep learning for robotics.  The International journal of robotics research, 37(4-5):405–420, 2018.
  * [71] Kushal Vangara, Michael C King, Vitor Albiero, Kevin Bowyer, et al.  Characterizing the variability in face recognition accuracy relative to race.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [72] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7464–7475, June 2023.
  * [73] Hongling Wang, Chengjin Zhang, Yong Song, Bao Pang, and Guangyuan Zhang.  Three-dimensional reconstruction based on visual slam of mobile robot in search and rescue disaster scenarios.  Robotica, 38(2):350–373, 2020.
  * [74] Christian Wankmüller, Maximilian Kunovjanek, and Sebastian Mayrgündter.  Drones in emergency response–evidence from cross-border, multi-disciplinary usability tests.  International Journal of Disaster Risk Reduction, 65:102567, 2021.
  * [75] Haiyu Wu, Vítor Albiero, KS Krishnapriya, Michael C King, and Kevin W Bowyer.  Face recognition accuracy across demographics: Shining a light into the problem.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1041–1050, 2023.
  * [76] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [77] Zhiheng Wu, Yue Lu, Xingyu Chen, Zhengxing Wu, Liwen Kang, and Junzhi Yu.  Uc-owod: Unknown-classified open world object detection.  In European Conference on Computer Vision, pages 193–210. Springer, 2022.
  * [78] Shizhou Zhang, Qi Zhang, Yifei Yang, Xing Wei, Peng Wang, Bingliang Jiao, and Yanning Zhang.  Person re-identification in aerial imagery.  IEEE Transactions on Multimedia, 23:281–291, 2020.
  * [79] Zhishuai Zhang, Cihang Xie, Jianyu Wang, Lingxi Xie, and Alan L Yuille.  Deepvoting: A robust and explainable deep network for semantic part detection under partial occlusion.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1372–1380, 2018.
  * [80] Xiaowei Zhao, Xianglong Liu, Yifan Shen, Yixuan Qiao, Yuqing Ma, and Duorui Wang.  Revisiting open world object detection.  arXiv preprint arXiv:2201.00471, 2022.
  * [81] Jiyang Zheng, Weihao Li, Jie Hong, Lars Petersson, and Nick Barnes.  Towards open-set object detection and discovery.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3961–3970, June 2022.
  * [82] Hongru Zhu, Peng Tang, Jeongho Park, Soojin Park, and Alan Yuille.  Robustness of object recognition under extreme occlusion in humans and computational models.  arXiv preprint arXiv:1905.04598, 2019.
  * [83] Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling.  Detection and tracking meet drones challenge.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7380–7399, 2021.
  * [84] Orr Zohar, Kuan-Chieh Wang, and Serena Yeung.  Prob: Probabilistic objectness for open world object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11444–11453, June 2023.

##  Appendix A Supplemental tables

Category | Descriptor | Values
---|---|---
Demographic | Age | Years integer.
| Weight | Pounds integer.
| Height | Inches integer.
| Gender | Male, Female
| Race | White Caucasian, Latino Hispanic, African Descent, Asian, South Asian, Middle Eastern,
|  | Pacific Islander.
| Clothes | Including no more than three dominant clothes, described by
|  | its relative location to the body (Upper, Lower, Both), its color (see color list),
|  | a pattern (Horizontal Stripes, Vertical Stripes, Both, Plain, Random), and a figure.
| Hat | Any form of head covering, described by color.
| Gloves | Described by color.
| Shoes | Described by color and type. Types are categorized in 6 classes, by their coverage
|  | and formality: Bare, Open Informal (_e.g_. sandals), Open Formal (_e.g_. heels),
|  | Close Informal (_e.g_. tennis), Close Formal (_e.g_. Oxford), and Boots.
| Facial Hair | Mustache, Beard.
| Hair | Described by color and length (Bald, Short, Medium, Long).
Environmental | Date | MM:DD format.
| Time | Integer, in 24hr format.
| Location | Lake, Forest, Field, Golf Course, Quarry, Farm, School, School (Nature), Paintball.
| Weather | Temperature in Fahrenheit, wind speed in MPH, and a word descriptor (_e.g_. ”Sunny”).
| Video | 5.4k video resolution, 30 fps, 5472 by 3078 pixels’ frames, and focal length of 8 mm.
| EV | String of 6 values, in increasing order of altitude and starting from the reference.
Table 1: Metadata information attached to every actor. Demographic category includes all descriptors that could aid in identification tasks, including outfit description. Descriptors | Colors
---|---
Clothes, Hat, Gloves, Shoes | Red, Orange, Yellow, Green, Cyan, Blue, Purple, Pink, White, Gray, Black, Brown.
Hair | Black, Blond, Sandy, Brown, Gray, White, Red, Green, Blue, Pink, Other.
Table 2: List of colors associated to different metadata descriptors. Distance | GSD [mm/pixel] | Face area [pixels]
---|---|---
10 | 4.242 | 1981
30 | 12.729 | 220
50 | 21.213 | 79
70 | 29.697 | 40
90 | 38.184 | 24
Table 3: Ground Sampling Distance (GSD) values at camera optical axis
position, as well as the amount of pixels to cover the area of a person’s face
assuming a rectangle area of 155mm by 230mm.


Herm}(k)\rightarrow{\rm U}(k)$ of the system (195) satisfies the assumption
$(1)$ and $(2)$ in the beginning of this subsection. Since (195) is seen as a
subsystem of (65), the proof follows from Lemma 7.2 and the definition of
$\nu(u_{\rm cat})$.

######  Remark 7.3.

Now we see that the chosen minus sign in the coefficients of linear systems
(195) or (65) is to ensure the monodromy relation (201), which is further to
ensure the compatibility with the chosen Gelfand-Tsetlin chain
$\mathfrak{u}(1)\subset\cdots\subset\mathfrak{u}(n-1)\subset\mathfrak{u}(n)$
as the upper left corners embeddings.

######  Remark 7.4.

There exists a family of integrable systems $\mathcal{F}(u)$ on
$\mathfrak{u}(n)\cong{\rm Herm}(n)$ defined by Mishchenko and Fomenko,
parametrized by $\widehat{\mathfrak{h}_{\rm reg}}(\mathbb{R})$, such that
$\mathcal{F}(u_{\rm cat})$ coincides with the Gelfand-Tsetlin system. More
generally, we expect that there exists a family of mutiplicative analog
$m\mathcal{F}(u)$ of $\mathcal{F}(u)$ on $U(n)^{*}\cong{\rm Herm}^{+}(n)$ such
that the Riemann-Hilbert-Birkhoff map $\nu(u)$ intertwines $\mathcal{F}(u)$
and $m\mathcal{F}(u)$ for the same $u\in\widehat{\mathfrak{h}_{\rm
reg}}(\mathbb{R})$.

###  7.2 Explicit formula of the Alekseev-Meinrenken diffeomorphisms

In [4], the authors introduced a distinguished diffeomorphism
$\Gamma_{AM}:{\rm Herm}(n)\rightarrow{\rm Herm}^{+}(n)$ compatible with the
Gelfand-Tsetlin systems. In this subsection, we point out the relation between
$\Gamma_{AM}$ and $\nu(u_{\rm cat})$, and use the expression of Stokes
matrices at a caterpillar point to derive an explicit expression of
$\Gamma_{AM}$.

####  7.2.1 The Alekseev-Meinrenken diffeomorphims

The Poisson manifolds ${\mathfrak{u}}(n)^{*}\cong{\rm Herm}(n)$ and ${\rm
U}(n)^{*}\cong{\rm Herm}^{+}(n)$ carry the structures of Gelfand–Tsetlin
integrable system [28] and multiplicative Gelfand-Tsetlin system [22]
respectively. The transpose map $T:A\mapsto A^{T}$ is an anti-Poisson
involution of ${\rm Herm}(n)$, as well as of ${\rm Herm}^{+}(n)$. The fixed
point set of the transpose map on ${\rm Herm}(n)$ is the set ${\rm Sym}(n)$ of
real symmetric $n$ by $n$ matrices, and the intersection ${\rm
Sym}_{0}(n)={\rm Sym}(n)\cup{\rm Herm}_{0}(n)$ is a union of Lagrangian
submanifolds of ${\rm Herm}_{0}(n)$. Actually the space ${\rm Sym}_{0}(n)$ has
$2^{\frac{n(n-1)}{2}}$ components.

######  Theorem 7.5.

[4] There exists a unique diffeomorphism $\Gamma_{AM}:{\rm
Herm}(n)\rightarrow{\rm Herm}^{+}(n)$ with the following three properties:

$(a)$ $\Gamma_{AM}$ intertwines the Gelfand-Tsetlin maps:
$\mu\circ\Gamma_{AM}=\lambda$;

$(b)$ $\Gamma_{AM}$ intertwines the Thimm’s torus actions on ${\rm
Herm}_{0}(n)$ and ${\rm Herm}^{+}_{0}(n)$;

$(c)$ For any connected component $S$ of ${\rm Sym}_{0}(n)\subset{\rm
Herm}(n)$, $\Gamma_{AM}(S)\subset S$.

In particular, the map $\Gamma_{AM}$ is a Poisson isomorphism.

####  7.2.2 Principal bundles and cross sections

######  Proposition 7.6.

(see e.g., [4] Proposition 2.1) The restriction of the Gelfand-Tsetlin map
$\lambda$ to ${\rm Herm}_{0}(n)$ defines a principal bundle

$\lambda:{\rm Herm}_{0}(n)\rightarrow\mathcal{C}_{0}(n)$

over the cone $\mathcal{C}_{0}(n)$ with structure group the Thimm’s torus
$T^{\frac{n(n-1)}{2}}$. It further restricts to a principal bundle

$\lambda:{\rm Sym}_{0}(n)\rightarrow\mathcal{C}_{0}(n)$

with discrete structure group
$T_{\mathbb{R}}^{\frac{n(n-1)}{2}}\cong(\mathbb{Z}_{2})^{\frac{(n-1)n}{2}}$.
Similarly for (the restriction of) the logarithmic Gelfand-Tsetlin map
$\mu:{\rm Herm}_{0}^{+}(n)\rightarrow\mathcal{C}_{0}(n)$ and $\mu:{\rm
Sym}_{0}^{+}(n)\rightarrow\mathcal{C}_{0}(n)$. Here ${\rm Sym}_{0}^{+}(n)$
denotes the intersection of ${\rm Herm}_{0}^{+}(n)$ with ${\rm Sym}(n)$.

Note that ${\rm Sym}_{0}(n)$ has $2^{\frac{n(n-1)}{2}}$ components, and the
"real part"
$T_{\mathbb{R}}^{\frac{n(n-1)}{2}}\cong(\mathbb{Z}_{2})^{\frac{(n-1)n}{2}}$ of
the Thimm’s torus action on ${\rm Sym}_{+}(n)$ relates the different connected
components. Any connected component $S$ of ${\rm Sym}_{0}(n)\subset{\rm
Herm}_{0}(n)$ can be understood as a cross section of the principal bundle
$\lambda:{\rm Herm}_{0}(n)\rightarrow\mathcal{C}_{0}(n)$. Therefore, a
principal $T^{\frac{n(n-1)}{2}}$-bundle map

$\Gamma:({\rm Herm}_{0}(n),\lambda)\rightarrow({\rm Herm}_{0}^{+}(n),\mu={\rm
log}\circ\lambda)$

is uniquely characterized by the image $\Gamma(S)\subset{\rm Herm}^{+}_{0}(n)$
of $S$. In particular, we can specify a bundle map $\Gamma_{0}$ by imposing
the real condition: $\Gamma_{0}(S)\subset S$ for a given connected component
$S$ of ${\rm Sym}_{0}(n)$. By the $T_{\mathbb{R}}^{n(n-1)/2}$-equivariance,
$\Gamma_{0}(S)\subset S$ holds true for all connected component $S$ of ${\rm
Sym}_{0}(n)$. Thus it gives the geometric interpretation of the conditions
$(a)$-$(c)$ characterising the Alekseev-Meinrenken diffeomorphism
$\Gamma_{AM}$ in Theorem 1.14. Actually, the map $\Gamma_{0}$ coincides with
the restriction of $\Gamma_{AM}$ on ${\rm Herm}_{0}(n)$. However, the
existence of a smooth extension of $\Gamma_{0}$ to ${\rm Herm}(n)$ is not
obvious.

Therefore, the maps $\Gamma_{AM}$ and $\nu(u_{\rm cat})$ are both Poisson
diffeomorphism intertwining the Gelfand-Tsetlin systems, while map a
Lagrangian submanifold $S\subset{\rm Sym}_{0}(n)$ to different cross sections
of the Gelfand-Tsetlin map $\mu$.

####  7.2.3 Phase transformations

Let $\theta:{\rm Herm}_{0}(n)\rightarrow T^{1}\times\cdot\cdot\cdot\times
T^{n-1}=T^{\frac{n(n-1)}{2}}$ be a map from ${\rm Herm}_{0}(n)$ to the
Gelfand-Tsetlin tours. Associated to $\theta$, there is a gauge transformation

$\mathcal{X}_{\theta}:{\rm Herm}_{0}(n)\rightarrow{\rm Herm}_{0}(n);\
A\mapsto{\theta(A)}\bullet A,$

where $\theta(A)\bullet A$ denotes the Thimm’s action of $\theta(A)$ on $A$,
see (69). For each $1\leq k\leq n-1$, let us denote the $T^{k}$-component of
$\theta$ by

$\theta^{(k)}(A)={\rm
diag}(e^{i\theta_{1}^{(k)}(A)},...,e^{i\theta_{k}^{(k)}(A)})\in T^{k},\ \ for\
any\ A\in{\rm Herm}_{0}(n).$

Then it follows from (98) that the Gelfand-Tsetlin action and angle
coordinates on ${\rm Herm}_{0}(n)$ change under the map $\mathcal{X}_{\theta}$
as

$\displaystyle\lambda^{(i)}_{j}(\mathcal{X}_{\theta}(A))=\lambda^{(i)}_{j}(A),\hskip
14.22636pt\psi^{(i)}_{j}(\mathcal{X}_{\theta}(A))=\psi^{(i)}_{j}(A)+\delta_{ik}\theta_{j}^{(k)}(A).$
(202)

####  7.2.4 Proof of Theorem 1.15

We have seen that the maps $\Gamma_{AM}$ and $\nu(u_{\rm cat})$ relate to each
other by a gauge transformation. Thus to prove Theorem 1.15, we only need to
construct explicitly the transformation. We decompose it into three steps.

Reformulation of the map $\nu_{\rm cat}$. From the expression in Proposition
3.22, we observe that for any $1<i\leq n$ the normalized connection matrix has
a decomposition

$\displaystyle\widetilde{C}\big{(}E_{i},\delta_{i}(A)\big{)}=D^{(i-1)}_{L}(A)\cdot{\rm
diag}(-a^{(i-1)}_{1},...,-a^{(i-1)}_{i-1},1,...,1)\cdot R^{(i)}(A)\cdot
D^{(i)}_{R}(A),$ (203)

where $D^{(i-1)}_{L}(A)$ is the $n$ by $n$ diagonal matrix

$\displaystyle
D^{(i-1)}_{L,kk}(A)=\frac{\prod_{v=1}^{i-1}\Gamma(1+\frac{\lambda^{(i-1)}_{v}-\lambda^{(i-1)}_{k}}{2\pi\iota})}{\prod_{v=1}^{i}\Gamma(1+\frac{\lambda^{(i)}_{v}-\lambda^{(i-1)}_{k}}{2\pi\iota})},\
\ for\ 1\leq k\leq i-1,\ \ \ and\ \ D^{(i-1)}_{L,kk}(A)=1,otherwise,$

$D^{(i)}_{R}(A)$ is the $n$ by $n$ diagonal matrix function

$\displaystyle
D^{(i)}_{R,kk}(A)=\frac{\prod_{v=1}^{i}\Gamma(1+\frac{\lambda^{(i)}_{v}-\lambda^{(i)}_{j}}{2\pi\iota})}{\prod_{v=1}^{i-1}\Gamma(1+\frac{\lambda^{(i-1)}_{v}-\lambda^{(i)}_{j}}{2\pi\iota})},\
for\ 1\leq k\leq i,\ \ \ and\ \ D^{(i-1)}_{R,kk}(A)=1,otherwise,$

and $R^{(i)}(A)$ is a $n$ by $n$ matrix with entries

$\displaystyle
R^{(i)}_{kj}(A)=\frac{e^{\frac{\lambda^{(i-1)}_{k}-\lambda^{(i)}_{j}}{4}}}{N_{j}^{(i)}{\rm
sinh}(\frac{\lambda^{(i-1)}_{k}-\lambda^{(i)}_{j}}{2})},\ \ for\ 1\leq k,j\leq
i,$ $\displaystyle R^{(i)}_{kk}(A)=1,\ for\ k=i+1,...,n,\hskip
8.53581ptand\hskip 8.53581ptR^{(i)}_{kj}(A)=0\ otherwise.$

Now let us denote by

$\displaystyle D^{(i)}(A):=D^{(i)}_{R}(A)\cdot D^{(i)}_{L}(A)\cdot{\rm
diag}(-a^{(i)}_{1},...,-a^{(i)}_{i},1,...,1)$ (204)

the multiplication of the three diagonal matrices. Then we can rewrite the map
$\widetilde{C}(u_{\rm cat})$ in Proposition 3.21 as

$(\overrightarrow{\prod_{i=1}^{n-1}}D^{(i)}R^{(i)})\cdot D^{(n)}_{R}.$

Since $D^{(n)}_{R}(A)$ commutes with $e^{A_{n}}$ (recall that $A_{n}={\rm
diag}(\lambda^{(n)}_{1},...,\lambda^{(n)}_{n})$), we see that

######  Lemma 7.7.

The Riemann-Hilbert-Birkhoff map $\nu(u_{\rm cat})$ can be rewritten as

$\displaystyle\nu(u_{\rm cat}):{\rm Herm}_{0}(n)\rightarrow{\rm
Herm}^{+}_{0}(n);\ A\mapsto\mathcal{T}(A)e^{A_{n}}\mathcal{T}(A)^{-1}$ (205)

where $\mathcal{T}(A):=D^{(1)}(A)R^{(1)}(A)\cdot\cdot\cdot
D^{(n-1)}(A)R^{(n-1)}(A)$.

The phase transformation of $\nu(u_{\rm cat})$. Based on the expression (202)
and Lemma 7.7, we have

######  Lemma 7.8.

There exists a unique map $\theta:{\rm Herm}_{0}(n)\rightarrow
T^{\frac{n(n-1)}{2}}$, such that for all $A\in{\rm Sym}_{0}$ with the angle
coordinates $\psi^{(i)}_{j}(A)=0$, the Gelfand-Tsetlin angle variables

$\displaystyle\psi^{(i)}_{j}(\nu_{\rm cat}\circ\mathcal{X}_{\theta}(A))=0,\ \
\ for\ all\ 1\leq i\leq n-1,\ 1\leq j\leq i.$ (206)

Proof. It follows from Lemma 7.7 that for any $\theta$

$\displaystyle\nu_{\rm cat}\circ\mathcal{X}_{\theta}(A):{\rm
Herm}_{0}(n)\rightarrow{\rm Herm}^{+}_{0}(n);\
\mathcal{T}(\mathcal{X}_{\theta}(A))e^{A_{d_{n}}}\mathcal{T}(\mathcal{X}_{\theta}(A))^{-1}$
(207)

where
$\mathcal{T}(\mathcal{X}_{\theta}(A)):=D^{(1)}(\mathcal{X}_{\theta}(A))R^{(1)}(\mathcal{X}_{\theta}(A))\cdot\cdot\cdot
D^{(n-1)}(\mathcal{X}_{\theta}(A))R^{(n-1)}(\mathcal{X}_{\theta}(A))$.

Following (204), $D^{(i)}(A)=D^{(i)}_{R}(A)\cdot D^{(i)}_{L}(A)\cdot{\rm
diag}(-a^{(i)}_{1},...,-a^{(i)}_{i},1,...,1)$ for any $1\leq i\leq n-1$. Note
that $R^{(i)}(A)$ only depends on the Gelfand-Tsetlin action variables, which
are preserved by the action $\mathcal{X}_{\theta}$, thus
$R^{(i)}(A)=R^{(i)}(\mathcal{X}_{\theta}(A))$. For the same reason, we have
$D^{(i)}_{L}(A)=D^{(i)}_{L}(\mathcal{X}_{\theta}(A))$ and
$D^{(i)}_{R}(A)=D^{(i)}_{R}(\mathcal{X}_{\theta}(A))$. For the angle functions
$\psi^{(i)}_{j}(A)$, by (202) we have
$\psi^{(i)}_{j}(\mathcal{X}_{\theta}(A))=\theta_{j}^{(i)}(A)+\psi^{(i)}_{j}(A)+\pi\iota$.
Therefore,

$\displaystyle
D^{(i)}(\mathcal{X}_{\theta}(A))=D^{(i)}_{R}(A)D^{(i)}_{L}(A)\cdot{\rm
diag}(e^{\theta_{1}^{(i)}(A)+\psi^{(i)}_{1}(A)+\pi\iota},...,e^{\theta_{i}^{(i)}(A)+\psi^{(i)}_{i}(A)+\pi\iota},1,...,1)$

Thus the $(j,j)$ entry of the above diagonal matrix is

$\displaystyle{\rm Arg}(D^{(i)}_{jj}(\mathcal{X}_{\theta}(A)))={\rm
Arg}(D^{(i)}_{R,jj}(A)D^{(i)}_{L,jj}(A))+\psi^{(i)}_{j}(A)+\pi\iota+\theta_{j}^{(i)}(A),\
j\leq i$ $\displaystyle{\rm Arg}(D^{(i)}_{jj}(\mathcal{X}_{\theta}(A)))=1,\
j>i.$

For any $A\in{\rm Sym}_{0}$ with the angle coordinates $\psi^{(i)}_{j}(A)=0$,
the condition ${\rm Arg}(D^{(i)}_{jj}(\mathcal{X}_{\theta}(A)))=0$, for all
$1\leq i\leq n-1$ and $1\leq j\leq i$, is equivalent to a linear system

$\theta_{j}^{(i)}(A)=-{\rm Arg}(D^{(i)}_{R,jj}(A)D^{(i)}_{L,jj}(A))-\pi\iota,\
\ \ for\ all\ 1\leq i\leq n-1,\ 1\leq j\leq i.$

Let us denote the solution of the linear system by $\theta:{\rm
Herm}_{0}(n)\rightarrow T^{\frac{n(n-1)}{2}}$.

From the above discussion, for any $A\in{\rm Sym}_{0}$ the matrix
$D^{(i)}(\mathcal{X}_{\theta}(A))R^{(i)}(\mathcal{X}_{\theta}(A))$ is real.
Together with (207), we see that the composed map $\nu(u_{\rm
cat})\circ\mathcal{X}_{\theta}$ restricts to a map from ${\rm Sym}_{0}$ to
${\rm Sym}_{0}^{+}$. If we impose further the angle coordinates
$\phi^{(i)}_{j}(A)=0$, then the angle variables $\phi^{(i)}_{j}$ of
$\nu(u_{\rm cat})\circ\mathcal{X}_{\theta}(A)$ is also zero. It finishes the
proof.

The explicit expression of $\Gamma_{AM}$. Now it is straightforward to write
down the map $\Gamma_{AM}$. First

$\Gamma_{AM}(A)=\nu(u_{\rm cat})(\mathcal{X}_{\theta}(A))=\widetilde{C}(u_{\rm
cat},\mathcal{X}_{\theta}(A))\cdot A_{n}\cdot\widetilde{C}(u_{\rm
cat},\mathcal{X}_{\theta}(A))^{-1},\ \ \forall\ A\in{\rm Herm}(n).$

Following the proof of Lemma 7.8, the phase transformation $\theta$ transforms
the gamma functions taking the form of $\Gamma(1+\frac{r}{2\pi\iota})$ for
$r\in\mathbb{R}$, in the expression of $\widetilde{C}(u_{\rm
cat},\mathcal{X}_{\theta}(A))$ for any $A\in{\rm Sym}_{0}$, to their real
parts. By the complex conjugate of gamma function and Euler’s reflection
formula

$\displaystyle\overline{\Gamma(z)}=\Gamma(\bar{z}),\
z\notin\mathbb{C}\setminus\\{0,-1,-2,...\\},$
$\displaystyle\Gamma(z)\Gamma(1-z)=\frac{\pi}{{\rm sin}(\pi z)},\hskip
8.53581ptz\notin\mathbb{Z},$

we get $|\Gamma(1+\frac{r}{2\pi\iota})|=\sqrt{\frac{2r}{2{\rm
sinh}(\frac{r}{2})}}$ for $r\in\mathbb{R}$. Thus replacing the terms
$\Gamma(1+\frac{r}{2\pi\iota})$ by their norms $\sqrt{\frac{2r}{2{\rm
sinh}(\frac{r}{2})}}$, we get the expression of the matrix
$\widetilde{C}\big{(}E_{k},\delta_{k}(\mathcal{X}_{\theta}(A))\big{)}$,

$\displaystyle\widetilde{C}\big{(}E_{k},\delta_{k}(\mathcal{X}_{\theta}(A))\big{)}_{ij}=\frac{e^{\frac{\lambda^{(k-1)}_{i}-\lambda^{(k)}_{j}}{4}}a^{(k-1)}_{i}}{N_{j}^{(k)}(\lambda^{(k-1)}_{i}-\lambda^{(k)}_{j})}\sqrt{\frac{H_{ij}^{(k)}\prod_{v=1,v\neq
i}^{k-1}{\rm
sinh}\big{(}\frac{\lambda^{(k-1)}_{v}-\lambda^{(k)}_{j}}{2}\big{)}\prod_{v=1,v\neq
j}^{k}{\rm
sinh}(\frac{\lambda^{(k-1)}_{i}-\lambda^{(k)}_{v}}{2})}{\prod_{v=1,v\neq
j}^{k}{\rm
sinh}(\frac{\lambda^{(k)}_{v}-\lambda^{(k)}_{j}}{2})\prod_{v=1,v\neq
i}^{k-1}{\rm sinh}(\frac{\lambda^{(k-1)}_{i}-\lambda^{(k-1)}_{v}}{2})}},$

for $1\leq i\leq k-1,1\leq j\leq k$, and

$\displaystyle\widetilde{C}\big{(}E_{k},\delta_{k}(\mathcal{X}_{\theta}(A))\big{)}_{kj}=\frac{e^{\frac{\lambda^{(k)}_{j}-\lambda^{(k-1)}_{k}}{4}}}{N_{j}^{(k+1)}}\sqrt{\frac{\prod_{v=1}^{k-1}{\rm
sinh}(\frac{\lambda^{(k-1)}_{v}-\lambda^{(k)}_{j}}{2})\prod_{v=1,v\neq
j}^{k}(\lambda^{(k)}_{v}-\lambda^{(k)}_{j})}{\prod_{v=1,v\neq j}^{k}{\rm
sinh}(\frac{\lambda^{(k)}_{v}-\lambda^{(k)}_{j}}{2})\prod_{v=1}^{k-1}(\lambda^{(k-1)}_{v}-\lambda^{(k)}_{j})}},\
\ \ for\ 1\leq j\leq k,$
$\displaystyle\widetilde{C}\big{(}E_{k},\delta_{k}(\mathcal{X}_{\theta}(A))\big{)}_{ii}=1,\
\ \ \ \ \ for\ k<i\leq n,$
$\displaystyle\widetilde{C}\big{(}E_{k},\delta_{k}(\mathcal{X}_{\theta}(A))\big{)}_{ij}=0,\
\ \ \ \ \ otherwise.$

Here $N_{j}^{(k)}$ and $H^{(k)}_{ij}$ are given by

$\displaystyle
N_{j}^{(k)}:=\sqrt{1+\sum_{l=1}^{k-1}\frac{|a_{l}^{(k-1)}|^{2}}{(\lambda^{(k-1)}_{l}-\lambda^{(k)}_{j})^{2}}},\
\ \ \ \ \ H^{(k)}_{ij}=\frac{\prod_{v=1,v\neq
j}^{k}(\lambda^{(k)}_{v}-\lambda^{(k)}_{j})\prod_{v=1,v\neq
i}^{k-1}(\lambda^{(k-1)}_{i}-\lambda^{(k-1)}_{v})}{\prod_{v=1,v\neq
i}^{k-1}(\lambda^{(k-1)}_{v}-\lambda^{(k)}_{j})\prod_{v=1,v\neq
j}^{k}(\lambda^{(k-1)}_{i}-\lambda^{(k)}_{v})}.$

Then Theorem 1.15 follows from the identity
$\sqrt{H_{ij}^{(k)}}=N_{j}^{(k)}(\lambda^{(k-1)}_{i}-\lambda^{(k)}_{j})/|a^{(k-1)}_{i}|$,
which follows from the character polynomial of the upper left $k+1$-th
submatrix of $A_{k}$.

####  7.2.5 Example: 2 by 2 cases.

Let $A=\left(\begin{array}[]{cc}a&b\\\ \bar{b}&c\end{array}\right)$ be a 2 by
2 Hermitian matrix. We will denote by

$\\{\lambda^{(1)}_{1}:=a,\lambda^{(2)}_{1},\lambda^{(2)}_{2},\psi^{(2)}_{1}:=b/{|b|}\\}$

the corresponding Gelfand-Tsetlin coordinates. Here
$\lambda^{(2)}_{1},\lambda^{(2)}_{2}$ are the eigenvalues of $A$. In this
case, the formula in Theorem 1.2 gives us

$\Gamma_{AM}:{\rm Herm}(2)\rightarrow{\rm Herm}^{+}(2);\
A=\left(\begin{array}[]{cc}a&b\\\
\bar{b}&c\end{array}\right)\mapsto\left(\begin{array}[]{cc}a^{\prime}&b^{\prime}\\\
\bar{b^{\prime}}&c^{\prime}\end{array}\right),$

where

$\left\\{\begin{array}[]{ll}a^{\prime}=e^{\lambda^{(1)}_{1}}\\\
b^{\prime}=e^{i\psi^{(2)}_{1}}\sqrt{e^{\lambda^{(1)}_{1}+\lambda^{(2)}_{1}}+e^{\lambda^{(1)}_{1}+\lambda^{(2)}_{2}}-e^{2\lambda^{(1)}_{1}}-e^{\lambda^{(2)}_{1}+\lambda^{(2)}_{2}}}\\\
c^{\prime}=e^{\lambda^{(2)}_{1}}+e^{\lambda^{(2)}_{2}}-e^{\lambda^{(1)}_{1}}.\end{array}\right.$

The above expression in $n=2$ case coincide with the one given in [4].
However, for general $n$, coordinate expressions for the Alekseev-Meinrenken
diffeomorphism or the Ginzburg-Weinstein maps were not known in the previous
works [1, 4, 11, 19, 26].

###  7.3 Boalch’s dual exponential maps are Poisson

####  7.3.1 Isomonodromy property and Poisson geometry nature of the closure
of Stokes matrices

######  Proposition 7.9.

For any $u\in U_{\rm id}\subset\mathfrak{h}_{\rm reg}(\mathbb{R})$, the
diffeomorphism given in Definition 3.29

$\Phi_{u_{\rm cat}}(u):{\rm Herm}(n)\cong{\mathfrak{u}}(n)^{*}\rightarrow{\rm
Herm}(n)\cong{\mathfrak{u}}(n)^{*};A\mapsto\Phi(u;A)$

is a Poisson isomorphism.

Proof. On the one hand, due to the Hamiltonian description of the isomonodromy
equation, the solution $\Phi(u;A)$ can be seen as a Hamiltonian flow. Thus for
any fixed $u,u^{\prime}\in U_{\rm id}$, the map

$\Phi_{u_{\rm cat}}(u^{\prime})\circ\Phi_{u_{\rm cat}}(u)^{-1}:{\rm
Herm}(n)\rightarrow{\rm Herm}(n);\Phi(u;A)\mapsto\Phi(u^{\prime};A)$

is Poisson. On the other hand, for any $u\in U_{\rm id}$ the map

$G(u):{\rm Herm}(n)\rightarrow{\rm
Herm}(n);A\mapsto\Big{(}\overrightarrow{\prod_{k=1,...,n}}(\frac{u_{k}}{u_{k+1}})^{\frac{A}{2\pi\iota}}\Big{)}\cdot
A\cdot\Big{(}\overrightarrow{\prod_{k=1,...,n}}(\frac{u_{k}}{u_{k+1}})^{\frac{A}{2\pi\iota}}\Big{)}^{-1},$

corresponding to a time $u$ flow of the Gelfand-Tsetlin system, is a Poisson
isomorphism. It follows that the map

$G(u^{\prime})\circ\Phi_{u_{\rm cat}}(u^{\prime})\circ\Phi_{u_{\rm
cat}}(u)^{-1}:{\rm Herm}(n)\rightarrow{\rm Herm}(n)$

is a Poisson isomorphism.

Now by Theorem 1.1 and Definition 3.10, we see that

$(G(u^{\prime})\circ\Phi_{u_{\rm
cat}}(u^{\prime}))(A)=G(u^{\prime})(\Phi(u^{\prime};A))\rightarrow A,\ \ as\
u^{\prime}\rightarrow u_{\rm cat}.$

Therefore, for fixed $u$ the Poisson maps $G(u^{\prime})\circ\Phi_{u_{\rm
cat}}(u^{\prime})\circ\Phi_{u_{\rm cat}}(u)^{-1}$ has the inverse map
$\Phi^{-1}_{u_{\rm cat}}(u)$ of $\Phi_{u_{\rm cat}}(u)$ as a limit when
$u^{\prime}\rightarrow u_{\rm cat}$. It finishes the proof.

Given the relation $\nu(u_{\rm cat})=\nu(u)\circ\Phi_{u_{\rm cat}}(u)$ (see
Definition 3.29), this proposition states that the closure of Stokes matrices
preserves the Poisson geometry nature. For example, it follows that the map
$\nu(u_{\rm cat})$ at the boundary point $u_{\rm cat}$ is Poisson.

####  7.3.2 Dual exponential maps are Poisson

Following Section 7.2.4, we have that $\Gamma_{AM}=\nu(u_{\rm
cat})\circ\mathcal{X}_{\theta}$, where $\mathcal{X}_{\theta}$ is the phase
transformation appearing in Section 7.2.4. Hence, by the relation $\nu(u_{\rm
cat})=\nu(u)\circ\Phi_{u_{\rm cat}}(u)$ (see Definition 3.29), we get for any
$u\in U_{\rm id}$

$\displaystyle\Gamma_{AM}=\nu(u)\circ\Phi_{u_{\rm
cat}}(u)\circ\mathcal{X}_{\theta}:{\rm Herm}(n)\rightarrow{\rm Herm}(n)^{+},$
(208)

As we have seen, the maps $\Gamma_{AM}$, $\Phi_{u_{\rm cat}}(u)$ and the phase
transformation $\mathcal{X}_{\theta}$ appearing in Section 7.2.4 are Poisson
due to some simple facts, i.e., intertwining action-angle variables, as
Hamiltonian flows and so on. Thus as a direct consequence of the relation
(208), we have that for any $u\in U_{\rm id}$ the map $\nu(u)$ is Poisson. In
this way, we give another proof of Boalch’s Theorem 1.10 on the Poisson
geometric nature of the Riemann-Hilbert-Birkhoff maps.

######  Remark 7.10.

We have used the fact that $\nu(u)$ is a diffeomorphism, due to Boalch, to
show $\Phi(u)$ is diffeomorphism. Thus we only give a new proof of the map
$\nu(u)$ is Poisson.

######  Definition 7.11.

The Riemann-Hilbert-Birkhoff map, associated to a representative
$T_{\sigma}(u)$ of $u_{0}\in\widehat{U_{\sigma}}$, is

$\displaystyle\nu^{\sigma}(u_{0}):{\rm Herm}(n)\rightarrow{\rm Herm}^{+}(n);\
A\mapsto
P_{\sigma}S_{-}^{\sigma}(u_{0},A){P_{\sigma}}^{-1}e^{[A]}P_{\sigma}S_{+}^{\sigma}(u_{0},A){P_{\sigma}}^{-1}.$
(209)

Similarly, using the results in Section 4.6, we can generalize the above
discussion for $u_{\rm cat}$ to any other boundary point, and then prove that
the map $\nu^{\sigma}(u_{0})$ is a Poisson isomorphism. In this way, we
generalize Theorem 1.10 to Theorem 1.11.

For any $u\in\mathfrak{h}_{\rm reg}(\mathbb{R})$ and $A\in{\rm Herm}(n)$, let
us denote by the matrix $\nu(u,A):=S_{-}(u,A)S_{+}(u,A)$, then as $0<u_{1}\ll
u_{2}\ll...\ll u_{n}$

$\displaystyle\Big{(}\overrightarrow{\prod_{k=1,...,n}}(\frac{u_{k-1}}{u_{k}})^{\frac{{\rm
log}(\delta_{k}(\nu(u,A)))}{2\pi\iota}}\Big{)}\cdot\nu(u,A)\cdot\Big{(}\overrightarrow{\prod_{k=1,...,n}}(\frac{u_{k-1}}{u_{k}})^{\frac{{\rm
log}(\delta_{k}(\nu(u,A)))}{2\pi\iota}}\Big{)}^{-1}\rightarrow\nu(u,\Phi(u;A)),$
(210)

###  7.4 A structure theorem of Stokes matrices via action-angle variables of
integrable systems

Recall that the Gelfand-Tsetlin variables
$\\{\lambda^{(k)}_{i},a^{(k)}_{i}\\}$ are defined on an open dense subset of
${\rm Herm}(n)$. Then

######  Definition 7.12.

For any $u\in U_{\rm id}\subset\mathfrak{h}_{\rm reg}(\mathbb{R})$, we define
the functions $\\{\lambda^{(k)}_{i}(u),a^{(k)}_{i}(u)\\}$ on (a proper open
dense subset of) ${\rm Herm}(n)$ such that their pull-back by $\Phi(u):{\rm
Herm}(n)\rightarrow{\rm Herm}(n)$ coincide with the Gelfand-Tsetlin variables.

Thus one gets a family of integrable systems by the pull-back of Gelfand-
Tsetlin system under the Poisson isomorphism $\Phi(u)$. Then
$\\{\lambda^{(k)}_{i}(u),a^{(k)}_{i}(u)\\}$ give rise to the action-angle
variables of the system. Furthermore, following from the properties of
Gelfand-Tsetlin action variables, we know that for any $u$, the functions
$\lambda^{(k)}_{i}(u)$ are smooth on an open dense subset and continuous to
the whole ${\rm Herm}(n)$, and the image of the collection of the functions
$\lambda^{(k)}_{i}(u)$ on ${\rm Herm}(n)$ coincides with the cone
$\mathcal{C}_{GZ}(\mathbb{R})$.

Theorem 1.2 can be seen as an expression of Stokes matrices $S_{\pm}(u_{\rm
cat},A)$ under the Gelfand-Tsetlin variables (provided we replace
$m^{(k)}_{i}$ by $a^{(k)}_{i}/{N_{i}^{(k+1)}}$). As an immediate corollary, we
have

######  Corollary 7.13.

For any $u\in U_{\rm id}$ and $A\in{\rm Herm}(n)$, the Stokes matrices
$S_{\pm}(u,A)$ are given by the formula in Theorem 1.2, provided we replace
the variables $\\{\lambda^{(k)}_{i},a^{(k)}_{i}\\}$ by the $u$-dependent
variables $\\{\lambda^{(k)}_{i}(u),a^{(k)}_{i}(u)\\}$.

Furthermore, it follows form the asymptotics of $\Phi(u;A)$ given in Theorem
1.1 that

######  Proposition 7.14.

For any $A\in{\rm Herm}(n)$, the function $\lambda^{(k)}_{i}(u;A)$ of $u$
approaches to the eigenvalue $\lambda^{(k)}_{i}(A)$ as $u_{1}\ll\cdots\ll
u_{n}$.

Thus Stokes matrices are completely described by a family of integrable
systems, and as $u\rightarrow u_{\rm cat}$, the action variables
$\lambda^{(k)}_{i}(u)$ of the associated integrable system continuously
extends, while the angle variables restricts to a fast spin on the
corresponding Liouville torus.

## References

  * [1] A. Alekseev, On Poisson actions of compact Lie groups on symplectic manifolds, J. Differential Geometry 45 (1997), 241-56.
  * [2] A. Alekseev, A. Berenstein, B. Hoffman and Y. Li, Langlands duality and Poisson-Lie duality via cluster theory and tropicalization, Selecta Math. (N.S.) 27 (2021), no. 4, paper no. 69.
  * [3] A. Alekseev, J. Lane and Y. Li, The $U(n)$ Gelfand-Tsetlin system as a tropical limit of Ginzburg-Weinstein diffeomorphisms, Phil. Trans. R. Soc. A 376: 20170428 (2018).
  * [4] A. Alekseev and E. Meinrenken, Ginzburg-Weinstein via Gelfand-Tsetlin, J. Differential Geom, 76 (2007) 1-34.
  * [5] A. Appel and S. Gautam, An explicit isomorphism between quantum and classical ${\rm sl}_{n}$, Transform. Groups, 25 (2020) 945–980.
  * [6] W. Balser, Explicit Evaluation of the Stokes’ Multipliers and Central Connection Coefficients for Certain Systems of Linear Differential Equations, Nath. Nachr. 138 (1988) 131-144.
  * [7] W. Balser, W.B. Jurkat, and D.A. Lutz, Birkhoff invariants and Stokes’ multipliers for meromorphic linear differential equations, J. Math. Anal. Appl. 71 (1979), 48-94.
  * [8] A. Berenstein and D. Kazhdan, Geometric and unipotent crystals II: From unipotent bicrystals to crystal bases quantum groups, Contemp. Math., vol. 433, Amer. Math. Soc., Providence, RI, 2007, pp. 13-88.
  * [9] A. Berenstein and A.N. Kirillov, Groups generated by involutions, Gel’fand-Tsetlin patterns and combinatorics of Young tableaux (Russian), Algebra i Analiz 7 (1995), no. 1, 92–152; translation in St. Petersburg Math. J. 7 (1996), no. 1, 77-127.
  * [10] P. Boalch, Stokes matrices, Poisson Lie groups and Frobenius manifolds, Invent. Math. 146 (2001), 479–506. no. 3, 479-506.
  * [11] P. Boalch, Symplectic manifolds and isomonodromic deformations, Adv. in Math. 163 (2001), 137–205.
  * [12] P. Boalch, G-bundles, isomonodromy and quantum Weyl groups, Int. Math. Res. Not. (2002), no. 22, 1129–1166.
  * [13] C. Bonnaf$\rm\acute{e}$, Cells and cacti, Int. Math. Res. Notices 19 (2016), 5775–5800.
  * [14] I. Cherednik, Lectures on Knizhnik-Zamolodchikov equations and Hecke algebras, Math. Soc. Japan Memoirs 1 (1998), 1–96.
  * [15] C. De Concini and C. Procesi, Wonderful models of subspace arrangements, Selecta Math. (N.S.) 1 (1995), no. 3, 459–494.
  * [16] G. Cotti, B. Dubrovin and D. Guzzetti, Isomonodromy deformations at an irregular singularity with coalescing eigenvalues, Duke Math. J. 168 (2019), no. 6, 967–1108.
  * [17] V. Drinfeld, On almost cocommutative Hopf algebras, Leningrad Math. J. 1 (1990), no. 2, 321–342.
  * [18] B. Dubrovin, Geometry of 2D topological field theories, Lecture Notes in Math, 1620 (1995).
  * [19] B. Enriquez, P. Etingof and I. Marshall, Comparison of Poisson Structures and Poisson-Lie dynamical r-matrices, Int. Math. Res. Not. 2005, no. 36, 2183–2198.
  * [20] B. Feigin, E. Frenkel, and V. Toledano Laredo, Gaudin model with irregular singularities, Adv. Math. 223 (2010), 873–948.
  * [21] G. Felder, Y. Markov, V. Tarasov and A. Varchenko, Differential equations compatible with KZ equations, Math. Phys. Anal. Geom. 3 (2000), 139-177.
  * [22] H. Flaschka and T. Ratiu, A Morse-theoretic proof of Poisson Lie convexity, Integrable systems and foliations/Feuilletages et systèmes intégrables (Montpellier, 1995), Progr. Math., vol. 145, Birkhäuser Boston, Boston, MA, 1997, pp. 49–71.
  * [23] B. Feigin, E. Frenkel, and L. Rybnikov, Opers with irregular singularity and spectra of the shift of argument subalgebra, Duke Math. J. 155 (2010), no. 2, 337–363.
  * [24] A. Fokas, A. Its, A. Kapaev and V. Novokshenov, Painlev${\acute{e}}$ Transcendents: The Riemann-Hilbert Approach, AMS Mathematical Surveys and Monographs, vol. 128, 2006.
  * [25] W. Fulton and R. MacPherson, A compactification of configuration spaces, Ann. Math. 139 (1994), 183–225.
  * [26] V. Ginzburg and A. Weinstein, Lie-Poisson structure on some Poisson-Lie groups, J. Amer. Math. Soc. 5:2 (1992), 445-53.
  * [27] M. Guest, A. Its and C. S. Lin, Isomonodromy Aspects of the $tt^{*}$ Equations of Cecotti and Vafa III: Iwasawa Factorization and Asymptotics, Commun. Math. Phys. 374 (2020), 923–973.
  * [28] V. Guillemin and S. Sternberg, The Gelfand-Cetlin system and quantization of the complex flag manifolds, J. Funct. Anal 52 (1983), 106–128.
  * [29] I. Halacheva, J. Kamnitzer, L. Rybnikov and A. Weekes, Crystals and monodromy of Bethe vectors, Duke Math. J. 169 (12) 2337-2419.
  * [30] J. Hartwig and N. Kingston, Gelfand-Tsetlin crystals, arxiv:2005.06639.
  * [31] M. Jimbo and T. Miwa, Monodromy preserving deformations of linear ordinary differential equations with rational coefficients II. Physica 2D, 407-448, (1981).
  * [32] M. Jimbo, T. Miwa and K. Ueno, Monodromy preserving deformations of linear differential equations with rational coefficients I, Physica 2D (1981), 306–352.
  * [33] M. Kashiwara, Crystallizing the q-analogue of universal enveloping algebra, Commun. Math. Phys. 133 (1990), 249-260.
  * [34] M. Kashiwara, On crystal bases of the q-analogue of universal enveloping algebras, Duke Math. J. 63 (1991), 465-516.
  * [35] D. Kazhdan, B, Kostant and S. Sternberg, Hamiltonian group actions and dynamical systems of Calogero type, Comm. Pure Appl. Math. XXXI, 481-507 (1978).
  * [36] I. Krichever, Isomonodromy equations on algebraic curves, canonical transformations and Whitham equations, Mosc. Math. J., 2(4): 717–752, 2002.
  * [37] J. Lin and X. Xu, Note on the Stokes matrices of certain linear systems of differential equations. In preparation.
  * [38] M. Loday-Richaud, Divergent series, summability and resurgence II. Simple and multiple summability, vol. 2154 of Lecture Notes in Mathematics, Springer, 2016.
  * [39] I. Losev, Cacti and cells, J. Eur. Math. Soc. (JEMS) 21 (2016), no. 2, 1729–1750.
  * [40] J.-H. Lu and A. Weinstein, Poisson Lie groups, dressing transformations, and Bruhat decompositions, Journal of Differential Geometry 31 (1990), 501 - 526.
  * [41] G. Lusztig, Canonical Bases Arising from Quantized Enveloping Algebras, Journal of the American Mathematical Society, Vol. 3, No. 2 (1990), 447-498.
  * [42] B. Malgrange and J.-P. Ramis, Fonctions multisommables, Ann. Inst. Fourier (Grenoble) 42 (1992), no. 1-2, 353–368.
  * [43] A.I. Molev, Gelfand-Tsetlin bases for classical Lie algebras, Handbook of algebra. Vol. 4, 109–170, Handb. Algebr., 4, Elsevier/North-Holland, Amsterdam, 2006.
  * [44] T. Nishinou, Y. Nohara and K. Ueda, Toric degenerations of Gelfand–Cetlin systems and potential functions. Adv. Math. 224(2), 648–706 (2010).
  * [45] F. W. J. Olver, D. W. Lozier, R. F. Boisvert and C. W. Clark, NIST Handbook of Mathematical Functions, Cambridge University Press, Cambridge, (2010).
  * [46] N. Reshetikhin, L. Takhtajan and L. Faddeev, Quantization of Lie groups and Lie algebras, Len. Math. J. 1 (1990), 193-225.
  * [47] L. Rybnikov, Argument shift method and Gaudin model (Russian), Funk. Analiz i Ego Pril. 40 (2006), no. 3, pp. 30–43; translation in Func. Anal. Appl. 40 (2006), No. 3, 188–199.
  * [48] D. Speyer, Schubert problems with respect to oscillating flags of stable rational curves, Alg. Geom. 1 (2014), no. 1, 14–45.
  * [49] W. Wasow, Asymptotic expansions for ordinary differential equations, Wiley Interscience, New York, 1976\.
  * [50] X. Xu, Irregular Riemann-Hilbert correspondence, Alekseev-Meinrenken r-matrices and Drinfeld twists, Int. Math. Res. Not. 23 (2018), 7420-7456.
  * [51] X. Xu, Stokes phenomenon, Gelfand-Tsetlin systems and relative Ginzburg-Weinstein linearization, Adv. Math. 338 (2018), 237-265.
  * [52] X. Xu, Stokes phenomenon and reflection equations, arXiv: 2006.03393.
  * [53] X. Xu, Drinfeld isomorphisms from quantum Stokes matrices, arXiv: 2012.15673.

School of Mathematical Sciences & Beijing International Center for
Mathematical Research, Peking University, Beijing 100871, China

E-mail address<EMAIL_ADDRESS>