# The MONET dataset: Multimodal drone thermal dataset
recorded in rural scenarios

Luigi Riz1 Andrea Caraffa1 Matteo Bortolon1 Mohamed Lamine Mekhalfi1 Davide
Boscani1
André Moura2 José Antunes2 André Dias2 Hugo Silva2 Andreas Leonidou3
Christos Constantinides3 Christos Keleshis3 Dante Abate3 Fabio Poiesi1
1Fondazione Bruno Kessler 2INESC TEC 3The Cyprus Institute

###### Abstract

We present MONET, a new multimodal dataset captured using a thermal camera
mounted on a drone that flew over rural areas, and recorded human and vehicle
activities. We captured MONET to study the problem of object localisation and
behaviour understanding of targets undergoing large-scale variations and being
recorded from different and moving viewpoints. Target activities occur in two
different land sites, each with unique scene structures and cluttered
backgrounds. MONET consists of approximately 53K images featuring 162K
manually annotated bounding boxes. Each image is timestamp-aligned with drone
metadata that includes information about attitudes, speed, altitude, and GPS
coordinates. MONET is different from previous thermal drone datasets because
it features multimodal data, including rural scenes captured with thermal
cameras containing both person and vehicle targets, along with trajectory
information and metadata. We assessed the difficulty of the dataset in terms
of transfer learning between the two sites and evaluated nine object detection
algorithms to identify the open challenges associated with this type of data.
Project page: https://github.com/fabiopoiesi/monet_dataset.

##  1 Introduction

Thermal image understanding enables localisation of objects that may not be
visible through traditional RGB cameras. ††This work was carried out within
the scope of the SHIELD project that received funding from the European
Union’s Joint Programming Initiative – Cultural Heritage, Conservation,
Protection and Use joint call. This can be useful in a variety of
applications, such as in surveillance and security, where illicit activities
typically occur overnight [26, 17], or in search and rescue [4], and military
operations, where targets can be easier to locate based on their emitted heat
rather than their cloth textures.

Figure 1: The MONET dataset is captured with a thermal camera mounted on a
multirotor drone. Three types of objects are annotated with bounding boxes:
green is person, magenta is vehicle, and blue is ignore. Outlined in grey are
the two recording sites: left-hand site is dirt-road and the right-hand site
is runway. MONET provides drone metadata (e.g. attitude, GPS, altitude)
timestamp aligned to each image. The map below is a screenshot from Google
Maps. Outlined in yellow are the two fields of view associated to the example
thermal images.

Object localisation in thermal images presents several challenges. Typically,
thermal imaging cameras have lower resolution than traditional RGB cameras,
hence distinguishing fine-scale details in images is difficult. Thermal
imaging cameras are susceptible to noise and ghost effects, such as
interference from other sources of heat or surface reflections. The appearance
of objects can vary depending on their temperature and emissivity, as well as
the ambient temperature and humidity, hence algorithms should be robust to
different environment conditions. Thermal images can be cluttered, thus
foreground objects may be indistinguishable from their background. Similarly
to RGB images, objects can also occlude each other, thus hindering multiple
object localisation. Objects of the same size and with the same temperature
(e.g. a boulder and a person), if captured from a distance, may appear with a
similar silhouette. In order to develop algorithms for object detection in
thermal images, it is necessary to have a large and diverse dataset of
annotated images. However, such datasets may be difficult to obtain, because
collecting large amounts of thermal data from drones and annotating them is
often costly.

Table 1: Relevant publicly available real-world datasets recorded from drones
outdoors. Keys. V: Visible. NIR: Near Infrared (841$nm$). RE: Red Edge
(717$nm$). LWIR: Long-Wave Infrared (7.5-13.5$\mu m$). The dataset name is
clickable and links to the dataset webpage.

Dataset attribute |  Campus [21] |  CARPK [9] |  VisDrone [29] |  UAVDT [8] |  BIRDSAI [2] |  AU-AIR [3] |  SeaDronesSee [25] |  HIT-UAV [23] | MONET
---|---|---|---|---|---|---|---|---|---
# images |  | 930K | 1.5K | 40K | 80K | 62K | 33K | 54K | 2.9K | 53K
# bounding boxes | 11M | 90K | 2.5M | 841K | 154K | 132K | 400K | 25K | 162K
# object trajectories | ✓ |  | ✓ | ✓ | ✓ |  | ✓ |  | ✓
Metadata |  |  |  | ✓ |  | ✓ | ✓ | ✓ | ✓
Categories | People | ✓ |  | ✓ |  | ✓ | ✓ | ✓ | ✓ | ✓
Vehicles | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ | ✓ | ✓
Animals |  |  |  |  | ✓ |  |  |  |
Sensor modality | V | V | V | V | LWIR | V | V+NIR+RE | LWIR | LWIR
Time/Weather | Day/Clear | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ | ✓ | ✓
Day/Foggy |  |  |  | ✓ |  |  |  |  |
Night/Clear |  |  | ✓ | ✓ | ✓ |  |  | ✓ | ✓
Scenario | Urban | ✓ | ✓ | ✓ | ✓ |  | ✓ |  | ✓ |
Rural |  |  | ✓ |  | ✓ |  |  |  | ✓
Maritime |  |  |  |  |  |  | ✓ |  |
Camera view | Fixed | ✓ | ✓ |  |  |  | ✓ |  |  |
Direction | top-down | top-down | varying | varying | varying | varying | varying | varying | varying
Drone attitude | Static | ✓ |  |  |  |  |  |  |  |
Moving |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Altitude [m] | 80 | 40 | n.a. | 10-70 | 60-120 | 5-30 | 5-260 | 60-130 | 20-130
Year | 2016 | 2017 | 2018 | 2018 | 2020 | 2020 | 2022 | 2022 | 2023

Publicly, we can find several drone datasets recorded with visible spectrum
cameras [21, 9, 8, 29, 3], instead thermal datasets are less popular. There
are some thermal datasets that are either annotated for single-object tracking
applications [1, 15] (one target max per frame is annotated), or they are
captured from static cameras resembling images captured from aerial vehicles
[19, 26]. Our focus is understanding scenes that can potentially contain
multiple objects and that are recorded from moving drones, but only few
datasets are available with such desired properties [2, 23]. One is BIRDSAI
[2], a long-wave thermal infrared (LWIR) dataset that contains nighttime
images of animals and humans in Southern Africa. Another one is HIT-UAV [23],
a LWIR dataset that contains both nighttime and daytime images of humans,
bicycles, and vehicles, captured by a drone in urban scenarios (schools,
parking lots, roads, playgrounds) flying between 60m to 130m altitude. The
SeaDroneSee dataset [25] also provides thermal images captured from drones,
but it includes scenes with humans and vehicles in water. Although water is a
challenging scenario, its challenges are different from terrain scenarios,
i.e. the structure of the environment is different, and background materials
can make targets indistinguishable if they emit the same heat. Moreover, drone
metadata (e.g. speed, altitude, drone and gimbal attitudes) is an important
piece of information because one can use it to retrieve the expected scale of
the targets to detect [18], or to calibrate the motion models of tracking
algorithms [13]. To this end, SeaDroneSee provides a comprehensive list of
metadata. HIT-UAV only provides information about altitude, camera
perspective, and a day/night flag. BIRDSAI does not provide metadata.

In this paper, we present MONET, a multimodal drone thermal dataset recorded
in rural scenarios that provides timestamp-aligned images and drone metadata
(see Fig. 1). MONET comprises approximately 53K frames, with about 162K
manually-annotated bounding boxes. The dataset includes two main target
categories, i.e. person and vehicle, plus a third category for a region to
ignore. Frames with targets contain 2.96 people and 1.33 vehicles on average.
Target bounding boxes are annotated with identities for multi-object tracking
applications. MONET’s metadata includes drone and gimbal attitude (pitch,
roll, and yaw), GPS, altitude, and speed (in the x, y, and z axes). The
dataset presents challenges such as sudden and fast camera motion, background
heat, large-scale variations, and different environmental structures. We
define two dataset scenarios, i.e. runway and dirt-road, which include scenes
recorded near a runway and in an agricultural land, respectively. Using the
Faster R-CNN detector [20], we analyse MONET’s challenges when training and
evaluating on the same scenario (e.g. runway to runway) and on different
scenarios (runway to dirt-road). Although the sensor is the same, experiments
show that training on one scenario and evaluating on the other leads to a
significant drop in performance. We evaluate nine object detectors and discuss
and analyse MONET’s challenges through qualitative results.

##  2 Related real-world datasets

We conduct a survey of related real-world datasets by analysing various
factors, including (i) object categories such as people, vehicles, and
animals, (ii) different sensor modalities, such as visible and infrared, (iii)
different scenarios, such as urban, rural, and maritime, (iv) different camera
views, such as fixed top-down and varying in different directions, and (v)
different drone attitudes, such as static and moving. The datasets were
recorded at various times of the day, with different weather conditions and at
different altitudes, and all of them provide annotations in the form of
bounding boxes (see Tab. 1).

The Campus dataset [21] comprises approximately 930K images with about 11M
bounding box instances of pedestrians, bicyclists, and vehicles. These targets
interact with each other within the Stanford University campus. The Campus
dataset was designed to facilitate multi-object tracking, activity
understanding, and trajectory forecasting. Object trajectories, along with
their IDs, were annotated. The images were captured in the visible spectrum
using a top-down camera during daytime from a multirotor drone hovering at an
altitude of about 80m.

The Car Parking Lot Dataset (CARPK) [9] comprises approximately 1.5K images
with about 90K bounding box instances of cars from four different parking lots
in urban scenarios. CARPK was designed for car counting, and no target IDs
were annotated. The images were captured in the visible spectrum using a top-
down camera during daytime from a multirotor drone flying at an altitude of
40m.

The VisDrone dataset [29] comprises approximately 40K images with about 2.5M
bounding box instances of pedestrians, vehicles, and bicycles, captured from
14 cities in China, between urban and rural scenarios. The images were
captured in the visible spectrum with arbitrary camera viewpoints during both
daytime and nighttime from a multirotor drone flying at different altitudes.
VisDrone provides object trajectory annotations, but no altitude information.

The UAVDT dataset [8] comprises 80K images with about 2.7K vehicles and 841K
bounding box vehicle instances, such as cars, trucks, and buses, from
different urban scenarios. The images were captured in the visible spectrum
from arbitrary viewpoints during both daytime and nighttime using a multirotor
drone at different altitudes. UAVDT provides object trajectory annotations and
sequence-level metadata information, including i) time/weather conditions
(daytime, nighttime, and fog), ii) flying altitude (low: 10-30m, medium:
30-70m, and high: above 70m), and iii) camera views (front, side, and bird).

The BIRDSAI dataset [2] comprises 62K images with about 120K animal and 34K
human bounding box instances from different national parks in Southern Africa.
BIRDSAI was designed for protected area monitoring to curb illegal activities
like poaching and animal trafficking. BIRDSAI provides object trajectory
annotations. The images were captured in the LWIR spectrum from arbitrary
viewpoints during nighttime by using a fixed-wing drone flying at altitudes
between 60m to 120m.

The AU-AIR dataset [3] comprises 33K images with about 132K bounding box
instances of people and vehicles in an urban scenario. This dataset was
designed for object detection tasks, hence target IDs are unavailable. The
images were captured in the visible spectrum with arbitrary camera viewpoints
during daytime by using a multirotor drone flying at altitudes between 5m to
30m. Unlike UAVDT, AU-AIR provides image-level metadata, which includes drone
speed, roll, pitch, yaw, altitude, latitude, and longitude. The camera is
fixed on the drone and it points in different directions during the flight.

The SeaDroneSee dataset [25] comprises 54K images with about 400K bounding box
instances of people and vehicles (boats) in different maritime scenarios.
SeaDroneSee was designed for search and rescue applications, specifically for
benchmarking multi-object tracking algorithms, hence object trajectories are
provided. Images were captured in both the visible and infrared spectrum by
using fixed-wing and multirotor drones flying at altitudes between 5m to 260m.
Similarly to AU-AIR, SeaDroneSee provides metadata logged at 10Hz, which
include drone speed, attitude, altitude, GPS, and gimbal pitch.

The HIT-UAV dataset [23] comprises 2.9K images with about 25K bounding box
instances of people and vehicles in urban scenarios. This dataset was designed
for object detection tasks, hence target IDs are unavailable. Images were
captured in the LWIR spectrum with arbitrary camera viewpoints during both
daytime and nighttime by using a multirotor drone flying at altitudes between
60m to 130m.

Unlike BIRDSAI, MONET includes metadata and the vehicle category instead of
the animal category, and is recorded from a multirotor drone as opposed to a
fixed-wing drone. Unlike HIT-UAV, MONET includes more metadata, and has
several more annotations that can also be used for multiple object-tracking
applications.

##  3 Hardware

###  3.1 Multirotor drone

We used a fully-customised multirotor drone, which was designed for automated
surveillance and detection of archaeological looting activities. The drone
includes the airframe, payload, propulsion, control, and communication
systems. The airframe is designed to be compact, lightweight, and capable of
carrying payloads up to 1.5kg. The propulsion system is composed of eight
motors and eight Electronic Speed Controllers, ensuring stability and
redundancy. It has four arms that support these eight motors. The control
system includes a high-frequency IMU, an accurate barometric altimeter, and an
external GPS and compass module. It supports several flight modes, including
manual, stabilised, heading hold, hovering, automated waypoint navigation,
return to home, auto take-off and auto-landing, and provides real-time
monitoring through an on-screen display. The communication system provides
real-time control and monitoring of the drone, and its payload.

###  3.2 Data acquisition system

The camera acquisition system consists of i) the camera WIRIS Security from
Workswell111https://workswell-thermal-camera.com/drone-security-thermal-
imaging-camera-night-vision-uav: last access: Apr. 2023. that features two
separate sensors, i.e. RGB and thermal, ii) the gimbal unit to physical hold
the camera on the drone, and iii) the Data Processing Unit. Fig. 2 illustrates
the architecture.

Figure 2: Data acquisition architecture that includes ground and drone system,
and WiFi communication between these two.

The WIRIS Security is provided with a proprietary SDK. In order to control the
camera, we developed a customised module that implements the WIRIS Security
control commands (through SDK) to the camera via Ethernet connection, which is
implemented in the drone ground station using ROS [22]. The RGB and thermal
images are received in the Data Processing Unit as RTSP video streams, and
published in ROS topics by using two independent nodes. The RGB sensor allows
up to 30x optical zoom, Full HD (1920$\times$1080) resolution, at a framerate
of 30Hz. The thermal sensor operates in the LWIR spectrum (7.5-13.5 $\mu$m)
featuring an 800$\times$600 resolution with -20∘C to 150∘C thermal
sensitivity. The camera includes an internal SSD drive with 256GB of storage
space, allows communication through Ethernet, USB and HDMI.

##  4 Dataset

We collected MONET in a rural area near the city of Nicosia, Cyprus, in mid-
December. The sequences were captured in the afternoon, evening and night. One
recording site is on a runway, which is property of The Cyprus Institute, the
other one is on agricultural lands. We name these sites as runway and dirt-
road, respectively. Fig. 1 shows the two sites.

###  4.1 Annotation procedure

Six people contributed to the annotation of MONET by using CVAT [7]. CVAT was
installed on a server and utilised via web browser. We created an account for
each annotator. A certain number of distinct sequences were assigned to each
annotator. We asked annotators to follow a set of guidelines: i) bounding
boxes should be drawn as tight as possible on the targets as long as the
object is clearly distinguishable from its background; ii) bounding box
interpolation across frames is allowed as long as each frame is checked to see
if the bounding boxes are correctly centred on targets; iii) brightness,
contrast and saturation can be adjusted through CVAT UI to make targets more
distinguishable; iv) if a target is partially occluded by another object or
indistinguishable from the background, its bounding box should be drawn based
on the best guess of the annotator and then flagged as occluded; v) the
annotation of a target should start when more than $\sim$30% of its pixels are
in the scene; vi) the annotation of a target should terminate when more than
$\sim$70% of its pixels are outside the scene; vii) if a target exits the
scene and then re-enters a new ID should be associated to it. Once the
annotations were completed, three people double-checked them to ensure they
were accurate and consistent with the guidelines. The Supplementary Material
contains annotation examples.

###  4.2 Bounding box categories and statistics

We annotated three types of targets: vehicle, person, and ignore. The bounding
boxes of vehicle include car-like objects, person is self-explanatory, and
ignore include the hangar location next to the runway. We decided to ignore
this location because other people and vehicles are often visible next to the
hangar, and we want to avoid data-driven detection algorithms to learn pattern
biases (e.g. people and vehicles often next to the hangar structure).
Therefore, we zeroed the regions of the images defined by the ignore bounding
boxes in order to avoid this bias during training. We provide the image data
in its original form if one wants to exploit this location or additional
targets for different purposes.

\begin{overpic}[width=253.6706pt]{figures/interpolation} \end{overpic} | \begin{overpic}[width=179.94939pt]{figures/occlusions} \end{overpic}
---|---

cat. | avg. | max | min
---|---|---|---
person | 2.96 | 6.00 | 1.00
vehicle | 1.33 | 4.00 | 1.00
ignore | 1.00 | 1.00 | 1.00

Figure 3: Bounding box annotation statistics of the MONET dataset. The top-
left graph highlights the number of interpolated and manually drawn bounding
boxes. The top-right graph highlights the number of occluded and non-occluded
bounding boxes: vehicle features 63 occluded bounding boxes (ignore is omitted
because always non-occluded). The table shows the average, maximum and minimum
of number of targets that can be found. The doughnut chart summarises the
percentage of person and vehicle bounding boxes.

Fig. 3 illustrates the bounding box annotation distributions of the whole
dataset. The left-hand side graph shows that the largest portion of bounding
boxes is manually drawn, while the others are (linearly) interpolated.
Interpolation is a feature of CVAT and is applied between consecutive frames.
While interpolation can be effective when annotations are made on videos
captured from static cameras, this is not the case when the camera is moving.
We only exploited interpolation occasionally because the motion of the drone
plus the motion of the targets cannot be modelled with a linear motion model.
The right-hand side graph shows that the largest portion of bounding boxes is
visible, while the others are occluded. The annotators flag a bounding box as
occluded when they deem the target was significantly occluded by another
target, or when the target was indistinguishable from the background, e.g. due
to interfering emitted temperatures. Although the target is indistinguishable
in some frames, we purposely annotated the bounding boxes (flagging them as
occluded) as it is a potential challenge if one’s use case is tracking. We
investigate the effect of training detectors with and without occluded
bounding box in Sec. 5.

###  4.3 Drone metadata

Together with the camera frames, MONET also includes drone metadata
information. As for the images, we logged the timestamp of each metadata
during dataset acquisition. Metadata was captured at about 40Hz on average.
Like [25], we used nearest-neighbour assignment between image and metadata
timestamps. Metadata includes the date in ISO 8601 format, the drone and
gimbal attitudes (pitch, roll, yaw), latitude, longitude, altitude and speed
(x, y, and z axes). Tab. 2 includes a detailed list of the metadata we
collected along with their minimum and maximum values.

Table 2: MONET metadata specifications indicating the units of each metadata
along with its minimum and maximum value.

Data | Unit | min. value | max. value
---|---|---|---
data | ISO 8601 | - | -
drone pitch | degrees | -90 | 90
drone roll | degrees | -90 | 90
drone yaw | degrees | 0 | 360
gimbal pitch | degrees | -40 | 90
gimbal roll | degrees | -45 | 45
gimbal yaw | degrees | -180 | 180
latitude | degrees | -90 | 90
longitude | degrees | -180 | 180
altitude | m | 0 | user defined
x-axis speed | cm/s | 0 | 2800
y-axis speed | cm/s | 0 | 2800
z-axis speed | m/s | 0 | 10

###  4.4 Examples of images and annotations

Fig. 4 shows examples of annotations in dirt-road and runway scenarios
recorded from the altitudes of 80m (a-c) and 100m (d). In particular, Fig. 4a
shows four person targets in the dirt-road scenario in normal visibility
conditions. Fig. 4b shows the same four targets as before, but when one target
is flagged as occluded: the small difference in the measured heat between
target and background makes them nearly indistinguishable. Fig. 4c shows a
similar case where a target is difficult to distinguish from the background,
but in this case this is due the vehicle’s heat behind the person. Lastly, we
show an example of an ignore region. This is the hangar hosting the drone
operators. We train our detection algorithms by zeroing the region of the
image defined by the ignore bounding box. See the project page for videos of
dirt-road and runway showing the annotations along with the aligned metadata.

\begin{overpic}[width=216.81pt,height=173.44534pt]{figures/example00.pdf} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{(a)}} \end{overpic} | \begin{overpic}[width=216.81pt,height=173.44534pt]{figures/example01.pdf} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{(b)}} \end{overpic}
---|---
\begin{overpic}[width=216.81pt,height=173.44534pt]{figures/example02.pdf} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{(c)}} \end{overpic} | \begin{overpic}[width=216.81pt,height=173.44534pt]{figures/example03.pdf} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{(d)}} \end{overpic}
Figure 4: Annotation examples showing (a,b) dirt-road and (c,d) runway
scenarios. Bounding boxes: green for person, magenta for vehicle, and blue for
ignore. Recording altitudes (a-c) 80m and (d) 100m. Dotted bounding boxes
indicate occlusions: (b) Example of occlusion where the target is
indistinguishable from the background. (c) Example of occlusion where the heat
from a vehicle makes the person indistinguishable. (d) Example of bounding box
labelled as ignore, enclosing the hangar area.

##  5 Experiments

###  5.1 Experimental setup

Scenes and settings. We split the dataset into two scenes: dirt-road and
runway, which include recordings of people activities nearby a runway and in
an agriculture land, respectively. Dirt-road is composed of 23.3K frames with
83.4K bounding box annotations, while runway is composed of 29.4K frames with
79.3K bounding box annotations. Each scene is divided into disjoint splits for
training, validation, and test. In Fig. 1 we can see that the structure of the
environment is different between these two scenes. So we conduct two sets of
experiments to analyse the challenges of MONET in terms of transfer learning
when object detection algorithms are trained and evaluated on different
scenes. Firstly, we focus on Faster R-CNN [20] and assess several transfer
learning configurations, including the combination of MONET and HIT-UAV [23]
data. Secondly, we compare the performance of nine popular object detectors,
aiming to understand MONET’s challenges. We use COCO evaluation procedure and
report AP, AP50, AP75, and AP per class [14]. Because each detector is
calibrated differently, setting a comparable detection confidence threshold is
impractical. Therefore, we evaluate all the detectors by using confidence
$10^{-8}$. This aspect is further discussed in the Supplementary Material.

Detectors. We use the MMDetection open source object detection toolbox [6] for
the following implementations: Faster R-CNN (2015) [20], SSD (2016) [16],
CornerNet (2018) [12], FCOS (2019) [24], DETR (2020) [5], Deformable DETR
(2021) [30], and VarifocalNet (2021) [28], while we use authors’
implementation for ObjectBox (2022) [27] and YOLOv8 (2023) [11]. We train all
these methods with the same data augmentations, and, where possible, with the
same backbones and optimisation parameters. Only for ObjectBox and YOLOv8 we
perform additional experiments with the data augmentations proposed by the
authors to investigate if they lead to different results. Please refer to the
Supplementary Material for the training configuration of each detector.

###  5.2 Quantitative results

Table 3: Evaluation of Faster R-CNN initialised with COCO trained weights.
Values are provided in percentage. Grey background indicates transfer learning
results. Keys: V: validation split. T: test split. †: additional pre-training
on top of COCO initialisation. w/o occ.: without bounding boxes with occlusion
flag.

Exp | Train/Eval | Detector | AP | AP50 | AP75 | class-AP
---|---|---|---|---|---|---
person | vehicle
1 |  dirt-road +runway | dirt-road V | 24.9 | 63.8 | 6.5 | 10.9 | 39.0
dirt-road T | 36.3 | 82.4 | 22.0 | 33.1 | 39.6
runway V | 44.5 | 90.2 | 38.9 | 31.8 | 57.2
runway T | 42.1 | 84.0 | 37.3 | 37.8 | 46.5
2 |  dirt-road +runway (w/o occ.) | dirt-road V | 25.1 | 67.8 | 7.8 | 13.6 | 36.6
dirt-road T | 36.3 | 81.3 | 21.9 | 32.1 | 40.5
runway V | 39.1 | 82.9 | 35.7 | 18.6 | 59.6
runway T | 47.6 | 88.5 | 46.0 | 43.2 | 51.9
3 |  HIT-UAV† +dirt-road +runway | dirt-road V | 28.4 | 70.1 | 9.3 | 16.3 | 40.4
dirt-road T | 39.1 | 85.0 | 29.1 | 32.1 | 46.0
runway V | 47.3 | 89.2 | 46.8 | 33.7 | 60.8
runway T | 46.2 | 87.7 | 43.6 | 43.5 | 48.8
4 | HIT-UAV | dirt-road V | 3.9 | 10.4 | 1.6 | 5.2 | 2.5
dirt-road T | 8.9 | 21.3 | 5.3 | 17.0 | 0.8
runway V | 28.7 | 65.5 | 19.5 | 27.6 | 29.9
runway T | 12.7 | 39.5 | 4.5 | 17.0 | 8.4
HIT-UAV V | 46.6 | 82.6 | 47.1 | 39.6 | 53.5
HIT-UAV T | 48.0 | 84.1 | 48.9 | 41.0 | 55.1
5 |  dirt-road +runway | HIT-UAV V | 6.7 | 17.5 | 3.4 | 10.3 | 3.1
HIT-UAV T | 7.1 | 18.5 | 4.3 | 11.0 | 3.3
6 | dirt-road | dirt-road V | 22.0 | 67.5 | 4.3 | 15.2 | 28.9
dirt-road T | 32.7 | 79.4 | 19.6 | 33.0 | 32.4
runway V | 20.4 | 42.2 | 18.2 | 40.5 | 0.3
runway T | 18.0 | 50.7 | 9.0 | 19.2 | 16.8
7 | runway | dirt-road V | 15.2 | 44.5 | 3.8 | 1.5 | 29.0
dirt-road T | 21.3 | 58.7 | 5.3 | 15.0 | 27.7
runway V | 38.8 | 82.5 | 35.7 | 19.2 | 58.4
runway T | 46.2 | 87.5 | 43.0 | 45.6 | 46.8

Transfer learning analysis. Tab. 3 reports different experiments of our
transfer learning analysis that we obtained by using the Faster R-CNN detector
[20]. We chose this detector because it is widely used in several benchmarks
[2, 25]. We train models that are pre-trained on COCO [14].

In Exp. 1 we combine dirt-road and runway training data, and evaluate the
performance on their respective validation and test splits. In addition to
observing that the validation split is more challenging than the test split,
the person category results to be the most difficult one to detect. This is
mainly due to the background heat that makes the targets difficult to
distinguish. In Exp. 2 we train without the bounding boxes that are flagged as
occluded (still evaluating with the bounding boxes flagged occluded). We can
observe that the results are similar between the two experiments. This
suggests that the use of bounding boxes flagged occluded appears to marginally
help the person class, while slightly affecting the vehicle class. Because
HIT-UAV contains both person and vehicle, in Exp. 3 we pre-train our detector
on HIT-UAV [23] (in addition to starting from the model pre-trained on COCO)
and then train with dirt-road+runway. HIT-UAV pre-training leads to improved
performance compared to Exp. 1. In Exp. 4, we evaluate the transfer learning
ability of the detector from HIT-UAV to both dirt-road and runway. Although
HIT-UAV and MONET’s categories and sensor modalities (thermal) are the same,
experiments show that this is a rather challenging setting, i.e. the detector
poorly generalises between these scenarios. Performances on runway are higher
than those on dirt-road. This can be due to the fact that the structure of the
environment of runway is more similar to that of HIT-UAV. We also report the
upper bound on HIT-UAV in Exp. 4. In Exp. 5, we can see that there is poor
transfer learning ability when training is on dirt-road+runway and evaluation
is on HIT-UAV. Compared to HIT-UAV upper bound, we can observe that the
performance gap in transfer learning is rather large. In Exp. 6 & 7 we report
the transfer learning experiments focused on MONET’s scenarios. Despite being
recorded with the same sensor, training on one scenario and testing on the
other leads to lower performances than the same scenario setting.

Table 4: Evaluation of several detectors initialised with COCO trained
weights. Values are provided in percentage. Grey background indicates transfer
learning results. Keys: F. R-CNN [20]: Faster R-CNN. Def. DETR [5]: Deformable
DETR [5]. †: with data augmentations of the original paper. Bold indicates
best, underline indicates second best.

Exp | Train/Eval | Detector | AP | AP50 | AP75 | class-AP
---|---|---|---|---|---|---
person | vehicle
1 |  dirt-road/ dirt-road | F. R-CNN [20] | 22.0 | 67.5 | 4.3 | 15.2 | 28.9
SSD [16] | 19.6 | 64.2 | 4.4 | 11.3 | 28.0
CornerNet [12] | 10.1 | 46.8 | 0.2 | 1.3 | 18.9
FCOS [24] | 14.7 | 55.2 | 0.4 | 3.1 | 26.3
DETR [5] | 12.5 | 44.1 | 0.7 | 0.7 | 24.3
Def. DETR [5] | 16.5 | 55.2 | 1.4 | 3.4 | 29.6
VarifocalNet [28] | 21.4 | 61.5 | 3.6 | 8.2 | 34.7
ObjectBox [27] | 26.4 | 72.5 | 5.7 | 16.4 | 36.5
YOLOv8 [11] | 25.1 | 64.9 | 5.9 | 11.0 | 39.3
ObjectBox† [27] | 31.4 | 68.1 | 19.3 | 15.4 | 47.4
YOLOv8† [11] | 33.3 | 76.0 | 15.6 | 22.2 | 44.4
2 |  runway/ dirt-road | F. R-CNN [20] | 15.2 | 44.5 | 3.8 | 1.5 | 29.0
SSD [16] | 21.9 | 47.0 | 17.6 | 1.6 | 42.1
CornerNet [12] | 18.9 | 35.6 | 22.7 | 0.2 | 37.5
FCOS [24] | 19.2 | 47.0 | 10.8 | 0.3 | 38.0
DETR [5] | 8.8 | 26.9 | 0.7 | 0.0 | 17.6
Def. DETR [5] | 8.0 | 31.3 | 0.3 | 0.3 | 15.8
VarifocalNet [28] | 19.3 | 47.1 | 14.9 | 1.2 | 37.5
ObjectBox [27] | 14.9 | 36.9 | 4.8 | 1.0 | 28.8
YOLOv8 [11] | 20.8 | 34.0 | 25.8 | 0.5 | 41.0
ObjectBox† [27] | 32.6 | 62.9 | 28.4 | 12.1 | 53.0
YOLOv8† [11] | 32.6 | 63.4 | 32.3 | 11.4 | 53.8
3 |  runway/ runway | F. R-CNN [20] | 38.8 | 82.5 | 35.7 | 19.2 | 58.4
SSD [16] | 42.6 | 85.1 | 40.9 | 25.2 | 60.0
CornerNet [12] | 39.6 | 76.4 | 37.1 | 20.2 | 58.9
FCOS [24] | 44.5 | 88.6 | 41.3 | 28.9 | 60.2
DETR [5] | 31.2 | 81.2 | 20.2 | 16.0 | 46.4
Def. DETR [5] | 44.1 | 94.7 | 39.9 | 28.8 | 59.5
VarifocalNet [28] | 49.5 | 92.4 | 46.3 | 37.1 | 61.9
ObjectBox [27] | 49.5 | 93.7 | 48.3 | 39.7 | 59.2
YOLOv8 [11] | 53.5 | 93.0 | 58.8 | 42.0 | 65.1
ObjectBox† [27] | 51.4 | 95.0 | 49.9 | 41.3 | 61.6
YOLOv8† [11] | 52.4 | 95.8 | 54.9 | 45.0 | 59.8
4 |  dirt-road/ runway | F. R-CNN [20] | 20.4 | 42.2 | 18.2 | 40.5 | 0.3
SSD [16] | 18.6 | 43.1 | 13.6 | 37.0 | 0.3
CornerNet [12] | 22.7 | 51.9 | 18.0 | 20.2 | 25.2
FCOS [24] | 19.0 | 53.4 | 9.4 | 32.1 | 5.9
DETR [5] | 13.9 | 48.0 | 3.5 | 19.0 | 8.8
Def. DETR [5] | 32.9 | 73.0 | 26.0 | 33.8 | 32.0
VarifocalNet [28] | 30.7 | 73.0 | 21.8 | 35.3 | 26.1
ObjectBox [27] | 27.2 | 59.7 | 19.8 | 39.0 | 15.4
YOLOv8 [11] | 30.7 | 60.2 | 28.3 | 38.6 | 22.7
ObjectBox† [27] | 32.4 | 63.2 | 29.7 | 34.7 | 30.1
YOLOv8† [11] | 27.8 | 55.4 | 24.8 | 34.9 | 20.7

Detector comparisons. Tab. 4 reports the comparisons amongst the different
detectors. Experiments were executed by including the bounding boxes flagged
occluded and by zeroing the areas marked with the ignore label.

In the same-scenario setting, YOLOv8 and ObjectBox are the best performing
ones. In the transfer learning setting, SSD and Def. DETR are the best
performing ones, while YOLOv8 and VarifocalNet consistently perform second
best. YOLOv8 consistently outperforms the other detectors in terms of AP75,
which indicates its superior ability in estimating the correct bounding box
sizes. In terms of AP50, results are mixed. Although the two settings were
captured with the same camera and in similar locations, we can observe that
object detection is generally rather challenging for all the detectors,
especially in the transfer learning setting. All detectors perform poorly in
dirt-road, in particular with the person category. Dirt-road is more
challenging because it was captured during daytime, where ground heat makes
human targets more difficult to distinguish than the vehicle targets. Except
Def. DETR, all the detectors perform poorly on the vehicle class in dirt-
road/runway. We believe that this is because the vehicles were captured from
much fewer viewpoints in dirt-road than in runway, thus affecting
generalisation when tested in runway. YOLOv8’s original data augmentation
strategies appears to be effective in dirt-road but less effective in runway,
suggesting that a more tailored design of data augmentation for this problem
could help improving the detection accuracy.

\begin{overpic}[width=108.405pt]{figures/qualitative_results/dirt_road/GT.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ground truth}} \end{overpic} | \begin{overpic}[width=108.405pt]{figures/qualitative_results/dirt_road/dirt_road.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{trained on dirt-road}} \end{overpic} | \begin{overpic}[width=108.405pt]{figures/qualitative_results/dirt_road/runway.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{trained on runway}} \end{overpic} | \begin{overpic}[width=108.405pt]{figures/qualitative_results/dirt_road/hit_uav.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{trained on HIT-UAV}} \end{overpic}
---|---|---|---
\begin{overpic}[width=108.405pt]{figures/qualitative_results/runway/GT.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ground truth}} \end{overpic} | \begin{overpic}[width=108.405pt]{figures/qualitative_results/runway/dirt_road.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{trained on dirt-road}} \end{overpic} | \begin{overpic}[width=108.405pt]{figures/qualitative_results/runway/runway.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{trained on runway}} \end{overpic} | \begin{overpic}[width=108.405pt]{figures/qualitative_results/runway/hit_uav.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{trained on HIT-UAV}} \end{overpic}
Figure 5: Qualitative results showing the performances of Faster R-CNN in the dirt-road (first row) and runway (second row) scenarios with different training datasets. Bounding boxes: green for person, magenta for vehicle. \begin{overpic}[width=216.81pt]{figures/qualitative_results/hit_uav/hit_uav.jpg} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{trained on HIT-UAV}} \end{overpic} | \begin{overpic}[width=216.81pt]{figures/qualitative_results/hit_uav/complete.jpg} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{trained on dirt-road+runway}} \end{overpic}
---|---
Figure 6: Qualitative results showing the performances of Faster R-CNN in a
scene from HIT-UAV with different training datasets.

###  5.3 Qualitative results

Fig. 5 shows examples of Faster R-CNN’s detections on dirt-road (first row)
and on runway (second row) produced with different training data. In the first
case, training on dirt-road and evaluating on dirt-road leads to satisfactory
results, when we train on runway the detector misses all the targets, and when
we train on HIT-UAV the detector detects some targets plus some false
positives. Conversely, in the second case, training on dirt-road and
evaluating on runway leads to good results, same as when we train on runway,
however when we train on HIT-UAV the detector produces several false positive
detections. We observed that one of the reasons for this is that HIT-UAV
contains several annotations of parked vehicles, which emits a low
temperature, see example in Fig. 6. Hence, the detector relates dark patterns
to vehicles, which are similar to the false positive detections in runway in
Fig. 5. Fig. 6 also shows that training on MONET and evaluating on HIT-UAV
leads to poor results. The Supplementary Material contains additional
qualitative results from the nine detectors evaluated.

##  6 Conclusions

We introduce MONET, a novel and challenging multimodal dataset for vision-
based object localisation in the thermal spectrum from drones. MONET was
collected with a drone that flew over rural areas and captured human and
vehicle activities. MONET comprises two scenarios in agricultural lands,
namely dirt-road and runway. We benchmarked nine state-of-the-art object
detection algorithms on MONET and found that they performed poorly due to the
large scale variation of the targets, and the background clutter caused by the
ground heat. The dirt-road scenario is more difficult than the runway scenario
as it contains several more of the above-mentioned challenges than runway. To
our knowledge, MONET is one of the few datasets in the thermal spectrum that
provides a large number of manually annotated frames with timestamp-aligned
metadata. Moreover, MONET includes bounding boxes with identities for each
target, making it suitable for multi-object tracking research. We hope that
MONET will foster further research, especially in the development of
multimodal solutions for object localisation that exploits knowledge from
metadata.

Limitations. Although we collected both RGB and thermal images and annotated
both of them, for now we will not include the RGB images in this release of
MONET due to still unaddressed privacy concerns. It is worth noting that only
a portion of the dataset contains RGB images captured in light, while the
others were captured in darkness.

## References

  * [1] A. Berg, J. Ahlberg, and M. Felsberg.  A thermal object tracking benchmark.  In AVSS, 2015.
  * [2] E. Bondi, R. Jain, P. Aggrawal, S. Anand, R. Hannaford, A. Kapoor, J. Piavis, S. Shah, L. Joppa, B. Dilkina, and M. Tambe.  Birdsai: A dataset for detection and tracking in aerial thermal infrared videos.  In WACV, 2020.
  * [3] I. Bozcan and E. Kayacan.  AU-AIR: A multi-modal unmanned aerial vehicle dataset for low altitude traffic surveillance.  In ICRA, 2020.
  * [4] C. Burke, P.R. McWhirter, J. Veitch-Michaelis, O. McAree, H.A.G. Pointon, S. Wich, and S. Longmore.  Requirements and limitations of thermal drones for effective search and rescue in marine and coastal areas.  Drones, 3(4), 2019.
  * [5] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [6] K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, Z. Zhang, D. Cheng, C. Zhu, T. Cheng, Q. Zhao, B. Li, X. Lu, R. Zhu, Y. Wu, J. Dai, J. Wang, J. Shi, W. Ouyang, C.C. Loy, and D. Lin.  MMDetection: Open mmlab detection toolbox and benchmark.  arXiv:1906.07155, 2019.
  * [7] CVAT.ai Corporation.  Computer Vision Annotation Tool (CVAT), 2022.
  * [8] D. Du, Y. Qi, H. Yu, Y. Yang, K. Duan, G. Li, W. Zhang, Q. Huang, and Q. Tian.  The unmanned aerial vehicle benchmark: Object detection and tracking.  In ECCV, 2018.
  * [9] M.R. Hsieh, Y.L. Lin, and W.H. Hsu.  Drone-based object counting by spatially regularized regional proposal network.  In ICCV, 2017.
  * [10] G. Jocher.  YOLOv5 by Ultralytics, 2020.
  * [11] G. Jocher, A. Chaurasia, and J. Qiu.  YOLO by Ultralytics, 2023.
  * [12] H. Law and J. Deng.  CornerNet: Detecting objects as paired keypoints.  In ECCV, 2018.
  * [13] S. Li and D.-Y. Yeung.  Visual object tracking for unmanned aerial vehicles: A benchmark and new motion models.  In AAAI, 2017.
  * [14] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, Dollar P., and C.L. Zitnick.  Microsoft COCO: Common Objects in Context.  In ECCV, 2014.
  * [15] Q. Liu, Z. He, X. Li, , and Y. Zheng.  PTB-TIR: A Thermal Infrared Pedestrian Tracking Benchmark.  2019\.
  * [16] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A.C. Berg.  SSD: Single shot multibox detector.  In ECCV, 2016.
  * [17] Y. Ma, X. Wu, G. Yu, and Y. Xu, Yand Wang.  Pedestrian detection and tracking from low-resolution unmanned aerial vehicle thermal imagery.  Sensors, 16(4), 2016.
  * [18] M. Messmer, B. Kiefer, and A. Zell.  Gaining scale invariance in UAV bird’s eye view object detection by adaptive resizing.  In ICPR, 2022.
  * [19] J. Portmann, S. Lynen, M. Chli, and R. Siegwart.  People detection and tracking from aerial thermal views.  In ICRA, 2014.
  * [20] S. Ren, K. He, R. Girshick, and J. Sun.  Faster R-CNN: Towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [21] A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese.  Learning social etiquette: Human trajectory understanding in crowded scenes.  In ECCV, 2016.
  * [22] Stanford Artificial Intelligence Laboratory et al.  Robotic Operating System, 2021.
  * [23] J. Suo, T. Wang, X. Zhang, H. Chen, W. Zhou, and W. Shi.  HIT-UAV: A high-altitude infrared thermal dataset for unmanned aerial vehicles.  arXiv:2204.03245, 2022.
  * [24] Z. Tian, C. Shen, H. Chen, and T. He.  Fcos: Fully convolutional one-stage object detection.  In ICCV, 2019.
  * [25] L.A. Varga, B. Kiefer, Martin Messmer, and Andreas Zell.  SeaDronesSee: A maritime benchmark for detecting humans in open water.  In WACV, 2022.
  * [26] Z. Wu, N. Fuller, D. Theriault, and M. Betke.  A thermal infrared video benchmark for visual analysis.  In CVPRW, 2014.
  * [27] M. Zand, A. Etemad, and M. Greenspan.  Objectbox: From centers to boxes for anchor-free object detection.  2022\.
  * [28] H. Zhang, Y. Wang, F. Dayoub, and N. Sunderhauf.  VarifocalNet: An IoU-aware dense object detector.  CVPR, 2021.
  * [29] P. Zhu, L. Wen, D. Du, X. Bian, Q. Hu, and H. Ling.  Detection and tracking meet drones challenge.  TPAMI, 44(11), 2022.
  * [30] X. Zhu, W. Su, L. Lu, B Li, X Wang, and J Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In ICLR, 2021.

Supplementary Material

##  Appendix A Introduction

We provide some additional material in support of the main paper. The content
is organised as follows:

  * •

In Sec. B we provide details about the training configuration of each detector
and describe each data augmentation we used.

  * •

In Sec. C we analyse the statistics of each dataset split for the dirt-road
and runway scenarios.

  * •

In Sec. D we provide examples of qualitative results taken from each detector
that we evaluated.

##  Appendix B Detector setups

In the main paper we have two sets of results that compare state-of-the-art
detectors. For both sets we use non-maximum suppression with IoU threshold 0.5
and we evaluate all the bounding boxes with confidence above $10^{-8}$. The
reason behind this choice is that the models are calibrated differently, hence
their output confidences can not be directly compared. Setting a low
confidence threshold is to minimally filter detector predictions, resulting in
a fairer comparison of their performance.

The first set compares nine detectors with training configurations that we set
as similar as possible. Tab. 5 reports the details of the chosen
configurations.

We train all the detectors with the same data augmentation strategy. Data
augmentations are applied in the following order: i) RandomCrop: This crops a
portion of the image with a size determined by randomly sampling two
independent values within the interval [0.8, 1.0] and by multiplying them by
the height and width of the original image; ii) Resize: This randomly resizes
the eventually cropped image between (600, 800) and (300, 400) while keeping
its original aspect ratio. iii) RandomHorizontalFlip: This randomly flips the
image horizontally with a probability of 0.7. iv) Padding: This is applied to
make all the images of the same size, i.e. (600, 800). v) Normalisation: This
involves normalising the image pixels with a mean of 126.225 and a standard
deviation of 73.338. Note that these normalisation factors differ from the
standard ones computed on ImageNet and were explicitly calculated for the
MONET dataset.

Table 5: Detectors setup. Keys: BS: Batch Size. LR: Learning Rate. CosAnn.:
Cosine Annealing.

Detector | Epochs | BS | Backbone | LR | Schedule | Optimiser |
---|---|---|---|---|---|---|---
F. R-CNN [20] | 20 | 24 | ResNet-50 | 1e-3 | CosAnn. | AdamW |
SSD [16] | 20 | 24 | VGG-16 | 1e-3 | CosAnn. | AdamW |
CornerNet [12] | 20 | 9 | HourglassNet-104 | 1e-4 | CosAnn | AdamW |
FCOS [24] | 20 | 24 | ResNet-50 | 1e-4 | CosAnn | AdamW |
DETR [5] | 50 | 24 | ResNet-50 | 5e-5 | CosAnn | AdamW |
Def. DETR [30] | 20 | 9 | ResNet-50 | 5e-5 | CosAnn | AdamW |
VarifocalNet [28] | 20 | 24 | ResNet-50 | 1e-4 | CosAnn | AdamW |
ObjectBox [27] | 20 | 24 | YOLOv5 v6.0 | 1e-2 | Warmup | SGD |
CosAnn |
YOLOv8 [11] | 20 | 24 | YOLOv8.0x | 1e-2 | Warmup | SGD |
CosAnn |
ObjectBox†[27] | 40 | 24 | YOLOv5 v6.0 | 1e-2 | Warmup | SGD |
CosAnn |
YOLOv8†[11] | 40 | 24 | YOLOv8.0x | 1e-2 | Warmup | SGD |
CosAnn |

The second set compares ObjectBox [27] and YOLOv8 [11] with their original
data augmentation. We use the † in the main paper to represent these setups.

The data augmentations we use for ObjectBox are applied in the following
order: i) Mosaic: This combines 4 images (600, 800) into a single image (1200,
1600). Padding is then applied to produce a squared image of size (1600,
1600); ii) RandomAffine: This applies translation and scale operations. The
scale factor is randomly sampled from [0.5, 1.5] while the independent
vertical and horizontals shifts are randomly sampled in the interval [-160,
160], i.e. using a maximum absolute fraction of 0.1. The image is then resized
to (800, 800); iii) Blur: This blurs the image using a random kernel size
sampled in the interval [3, 7] with a probability of 0.1; iv) MedianBlur: This
blurs the image using a median filter with random aperture linear size sampled
in the interval [3, 7] with a probability of 0.1; v) RandomHSV: This firstly
converts the image to HSV colorspace. Then, three scalars are sampled with the
intervals [0.985, 1.015], [0.3, 1.7], and [0.6, 1.4], which are used to
multiply the original values of Hue, Saturation and Value, respectively.
Lastly, the image is converted back to RGB colorspace; vi)
RandomHorizontalFlip: This randomly flips the image horizontally with a
probability of 0.5.

The data augmentations we use for YOLOv8 are applied in the following order:
i) Mosaic: This combines 4 images (600, 800) into a single image (1200, 1600).
Padding is then applied to produce a squared image of size (1600, 1600); ii)
MixUp: This averages two mosaic images with a probability of 0.15; iii)
RandomAffine: This applies translation and scale operations. The scale factor
is randomly sampled from [0.1, 1.9] while vertical and horizontals shifts are
independently randomly sampled in the interval [-160, 160], i.e. using a
maximum absolute fraction of 0.1. The image is then resized to (800, 800); iv)
Blur: This blurs the image using a random kernel size sampled in the interval
[3, 7] with a probability of 0.01; v) MedianBlur: This blurs the image using a
median filter with a random aperture linear size sampled in the interval [3,
7] with a probability of 0.01; vi) CLAHE: This applies Contrast Limited
Adaptive Histogram Equalisation with probability 0.01; vii) RandomHSV: This
firstly converts the image to HSV colorspace. Then, three scalars are sampled
in the intervals [0.985, 1.015], [0.3, 1.7], and [0.6, 1.4], which are used to
multiply the original values of Hue, Saturation and Value, respectively.
Lastly, the image is converted back to RGB colorspace; viii)
RandomHorizontalFlip: This randomly flips the image horizontally with a
probability of 0.5.

##  Appendix C Additional dataset statistics

Figs. 7, 8, and 9, show the statistics of train, validation, and test splits
of dirt-road, respectively, while Figs. 10, 11, and 12, show the statistics of
train, validation, and test splits of runway, respectively. The statistics
include i) the histogram of the bounding box instances, ii) examples of
bounding boxes randomly sampled from the ground truth, iii) the distribution
of the bounding box locations over the image plane, and iv) the distribution
of the bounding box sizes as a function of the width and height. These figures
are generated with the software provided with YOLOv5 [10] and ObjectBox [27].
It is interesting to observe the difference in bounding box sizes between
dirt-road and runway splits.

##  Appendix D Additional qualitative results

Because the detectors are calibrated differently, it is unfair to apply the
same confidence threshold to visualise the results. Therefore, we choose a
different threshold for each detector that corresponds to the maximum between
$\gamma$ and 0.10, where $\gamma$ is the confidence value of the true positive
detection with lowest confidence value in a given frame. This approach allows
us to visualise all detected targets, but it may result in more false alarms.

Fig. 13 shows the qualitative results of the different detectors when their
models are trained on dirt-road and evaluated on dirt-road. We can observe
that this scenario is very challenging because all the detectors fail to
detect all the person targets. The vehicle is accurately detected by all the
detectors except for CornerNet.

Fig. 14 shows the qualitative results of the different detectors when their
models are trained on runway and evaluated on dirt-road. We can observe that
SSD is the only detector that can detect some person targets. All the others
either produce false alarms or do not detect any person targets. Like before,
the vehicle is accurately detected by all the detectors except for CornerNet.

Fig. 15 shows the qualitative results of the different detectors when their
models are trained on runway and evaluated on runway. Unlike before, all the
targets are correctly detected in this setting. Moreover, we can observe that
the confidence value of each detector is rather different from each other.

Fig. 16 shows the qualitative results of the different detectors when their
models are trained on dirt-road and evaluated on runway. We can observe that
Deformable DETR is the best performing one, followed by VarifocalNet. The most
noisy one resulted to be CornerNet.

Figure 7: Bounding box ground-truth statistics of dirt-road train split. The top-left figure shows the histogram of bounding box instances. The top-right figure shows 1K examples of randomly sampled bounding boxes. The bottom-left figure shows the distribution of bounding box locations over the image plane. The bottom-right figure shows the distribution of bounding box sizes. Figure 8: Bounding box ground-truth statistics of dirt-road validation split. The top-left figure shows the histogram of bounding box instances. The top-right figure shows 1K examples of randomly sampled bounding boxes. The bottom-left figure shows the distribution of bounding box locations over the image plane. The bottom-right figure shows the distribution of bounding box sizes. Figure 9: Bounding box ground-truth statistics of dirt-road test split. The top-left figure shows the histogram of bounding box instances. The top-right figure shows 1K examples of randomly sampled bounding boxes. The bottom-left figure shows the distribution of bounding box locations over the image plane. The bottom-right figure shows the distribution of bounding box sizes. Figure 10: Bounding box ground-truth statistics of runway train split. The top-left figure shows the histogram of bounding box instances. The top-right figure shows 1K examples of randomly sampled bounding boxes. The bottom-left figure shows the distribution of bounding box locations over the image plane. The bottom-right figure shows the distribution of bounding box sizes. Figure 11: Bounding box ground-truth statistics of runway validation split. The top-left figure shows the histogram of bounding box instances. The top-right figure shows 1K examples of randomly sampled bounding boxes. The bottom-left figure shows the distribution of bounding box locations over the image plane. The bottom-right figure shows the distribution of bounding box sizes. Figure 12: Bounding box ground-truth statistics of runway test split. The top-left figure shows the histogram of bounding box instances. The top-right figure shows 1K examples of randomly sampled bounding boxes. The bottom-left figure shows the distribution of bounding box locations over the image plane. The bottom-right figure shows the distribution of bounding box sizes. \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/GT.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ground truth}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-faster_rcnn.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{Faster R-CNN}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-ssd.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{SSD}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-cornernet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{CornerNet}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic}
---|---|---|---
\begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-fcos.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{FCOS}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-detr.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{DETR}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-deformable_detr.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{Deformable DETR}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-vfnet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{VarifocalNet}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic}
\begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-objectbox-mmdet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ObjectBox}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-yolo-mmdet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{YOLOv8}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-objectbox.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ObjectBox\textsuperscript{\textdagger}}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2d/d2d-yolo.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{YOLOv8\textsuperscript{\textdagger}}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic}
Figure 13: Qualitative results of the different detectors trained on dirt-road and evaluated on dirt-road. Bottom right value in each image represents the conference threshold adopted at inference time. Bounding boxes: green for person, magenta for vehicle. Recording altitude: 80m. \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/GT.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ground truth}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-faster_rcnn.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{Faster R-CNN}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-ssd.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{SSD}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-cornernet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{CornerNet}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic}
---|---|---|---
\begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-fcos.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{FCOS}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-detr.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{DETR}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-deformable_detr.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{Deformable DETR}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-vfnet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{VarifocalNet}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic}
\begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-objectbox-mmdet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ObjectBox}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-yolo-mmdet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{YOLOv8}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-objectbox.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ObjectBox\textsuperscript{\textdagger}}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2d/r2d-yolo.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{YOLOv8\textsuperscript{\textdagger}}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic}
Figure 14: Qualitative results of the different detectors trained on runway and evaluated on dirt-road. Bottom right value in each image represents the conference threshold adopted at inference time. Bounding boxes: green for person, magenta for vehicle. Recording altitude: 80m. \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/GT.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ground truth}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-faster_rcnn.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{Faster R-CNN}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.99}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-ssd.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{SSD}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.96}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-cornernet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{CornerNet}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.56}} \end{overpic}
---|---|---|---
\begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-fcos.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{FCOS}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.61}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-detr.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{DETR}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.97}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-deformable_detr.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{Deformable DETR}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.93}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-vfnet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{VarifocalNet}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.85}} \end{overpic}
\begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-objectbox-mmdet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ObjectBox}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.88}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-yolo-mmdet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{YOLOv8}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.78}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-objectbox.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ObjectBox\textsuperscript{\textdagger}}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.86}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/r2r/r2r-yolo.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{YOLOv8\textsuperscript{\textdagger}}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.82}} \end{overpic}
Figure 15: Qualitative results of the different detectors trained on runway and evaluated on runway. Bottom right value in each image represents the conference threshold adopted at inference time. Bounding boxes: green for person, magenta for vehicle. Recording altitude: 82m. \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/GT.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ground truth}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-faster_rcnn.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{Faster R-CNN}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-ssd.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{SSD}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-cornernet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{CornerNet}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic}
---|---|---|---
\begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-fcos.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{FCOS}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-detr.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{DETR}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-deformable_detr.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{Deformable DETR}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-vfnet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{VarifocalNet}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.46}} \end{overpic}
\begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-objectbox-mmdet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ObjectBox}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-yolo-mmdet.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{YOLOv8}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-objectbox.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{ObjectBox\textsuperscript{\textdagger}}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic} | \begin{overpic}[width=104.07117pt]{figures/qualitative_results/supplementary/d2r/d2r-yolo.png} \put(2.0,3.0){\color[rgb]{.75,1,0}\footnotesize{YOLOv8\textsuperscript{\textdagger}}} \put(85.0,3.0){\color[rgb]{.75,1,0}\footnotesize{0.10}} \end{overpic}
Figure 16: Qualitative results of the different detectors trained on dirt-road
and evaluated on runway. Bottom right value in each image represents the
conference threshold adopted at inference time. Bounding boxes: green for
person, magenta for vehicle. Recording altitude: 82m.


# DAOS as HPC Storage: Exploring Interfaces

Adrian Jackson  The University of Edinburgh, United Kingdom
<EMAIL_ADDRESS>Nicolau Manubens  European Centre for Medium-Range
Weather Forecasts, Germany
<EMAIL_ADDRESS>
##  I Introduction

Reading and writing data to and from data storage has long been a bottleneck
for high performance computing (HPC). The hardware used for data storage
exhibits performance that is generally at least an order of magnitude lower
than that of volatile memory or processing hardware. Applications often limit
the amount of I/O undertaken on HPC systems to reduce this cost, but recently
there has been a rise in application categories where ingestion or production
of large amounts of data is common, machine learning being an obvious example.
As HPC systems increase in size, reaching Exascale levels and beyond, and a
subset of applications require ever larger amounts of I/O bandwidth or
metadata performance, there is a significant challenge to improve the
performance of the data storage technologies employed for I/O operations.

Object storage technology has gained significant traction in the web services
and cloud markets for data that does not necessarily fit into standard
filesystem structures or require more storage than databases traditionally
provide. However, even the object stores that are used in scientific
computing, such as Ceph [18], are yet to scale to the I/O performance required
for large HPC deployments. The level of performance required for exascale
class systems is demonstrated by the recent Frontier system deployment[19],
where a 700PB, 5TB/s, Lustre filesystem is providing long term storage.

However, object stores have the potential to address long-standing performance
and scalability issues associated with POSIX filesystems for large-scale
parallel I/O applications, such as applications using large numbers of small
data files (KiBs up to a few MiBs) that can severely stress the metadata
functionality of implementations such as Lustre and Spectrum Scale. Such
issues can stem from constraints imposed by POSIX file system semantics,
namely metadata prescriptiveness (per-file creation date, last-access date,
permissions, etc.), and excessive consistency assurance[1]. Object storage
technology can offer less restrictive semantics to POSIX file systems and can
be unaffected by existing file system implementations or standards.

The Distributed Asynchronous Object Store (DAOS)[2] is a high-performance
object store which features full user-space operation, with a RAFT-based
consensus algorithm for distributed, transactional indexing, and byte-
addressable access to NVM devices. It has demonstrated strong I/O performance
as it comes to maturity, with high rankings in the I/O 500 benchmark[3] an
indicator that is can scale to high metadata operation and I/O bandwidth
rates. Previous research has demonstrated DAOS can provide high performance
and resiliency for HPC applications[12][15][20]

However, the exploitation of object stores like DAOS for HPC I/O brings a
number of usability and functionality questions that are yet to be fully
explored. DAOS can be accessed in many different ways, and with a range of
configurations, that bring a range of choices for both users and operators of
such as system. In this work progress we discuss our investigation of the
performance characteristics of the different access methodologies, and some of
the configurations, that DAOS provides, and evaluate the best options for
applications to utilise.

We present performance of standard I/O benchmarks that capture bulk I/O access
patterns, using DAOS deployed on non-volatile storage hardware. A research HPC
system containing nodes with 3 TiB of Intel’s Optane Data Centre Persistent
Memory Modules (DCPMMs) is used for benchmarking.

##  II DAOS

Distributed Asynchronous Object Store (DAOS) is an open-source object store
designed for massively distributed non-volatile data storage. Originally based
on exploiting Intel’s Optane memory, backed by NVMe storage devices, it has
recently been expanded to support systems where Optane is not present,
primarily by storing the metadata it holds in volatile memory backed by NVMe
devices, rather than directly in Optane non-volatile memory.

DAOS provides a low-level key-value storage interface on top of which other
higher-level APIs, also provided by DAOS, are built, include a FUSE-based
filesystem interface and custom interfaces for HDF5, MPI-I/O, and other common
data storage formats/approaches. DAOS includes features such as transactional
and non-blocking I/O, fine-grained I/O operations, end-to-end data integrity,
and advanced data protection. The OpenFabrics Interfaces (OFI) library is used
for low-latency communications over a wide range of high performance network
back-ends. It also supports object classes that range from S1 through to SX
which enable objects to be distributed across DAOS engines in a similar manner
to Lustre file striping.

DAOS enables a range of different methods for I/O from applications. The
lowest level, requiring the most application adaption, is using key-value and
array APIs (provided by libdaos) directly. There are also the options to use
DFS (DAOS filesystem), either using libdfs, an API for file accesses and I/O
on DAOS, or through DFuse (DAOS Fuse) which provides a user space filesystem
mount point for the DAOS object store. libdfs and DFuse both allow
applications to continue using file I/O whilst exploiting DAOS storage,
whereas the DAOS API requires object store functionality to be added to
applications.

##  III Methodology

To assess the performance of DAOS, different I/O workloads have been generated
and run in a research HPC system using the well-known IOR[14] benchmark.

For the tests in this analysis, the IOR benchmark has been run in both easy
and hard modes, easy being a file-per-process, and hard being a single shared-
file. IOR is invoked with a set of parameter values which instruct each client
process to perform a set of I/O operations (transfers), to write and read its
full data size. This is with the intent to assess the performance of a
hypothetical optimised parallel application which is designed to minimise the
number of I/O operations interacting with the storage and perform contiguous
reads or writes.

IOR was originally designed to test performance of file systems, and therefore
it can be used directly with its default POSIX back-end to test DAOS Fuse
mounts. However, IOR also includes multiple additional back-ends that can be
enabled to operate with the above-mentioned APIs, including libdaos’ byte-
array API, DFS, MPI-I/O, and HDF5. In this analysis, the DFS, MPI-I/O and HDF5
back-ends have been employed.

##  IV Results

The benchmarks we present have been conducted using the NEXTGenIO system[16],
which is a research HPC system composed of dual-socket nodes with Intel Xeon
Cascade Lake processors. Each socket has six 256 GiB first-generation Intel’s
Optane DCPMMs configured in AppDirect interleaved mode. We benchmarked using 8
server nodes with 2 DAOS engines per server node.

Figures 1 and 2 present read and write bandwidth for the file-per-process and
shared-file IOR test cases. We can see from the graphs that both the object
class (i.e. S1, S2, or SX) and the access mechanisms (i.e. DAOS, HDF5, or
MPI-I/O) impact the achieved performance.

(a) Read

(b) Write

Figure 1: IOR: File-per-process

It is clear that for the file-per-process scenario, having a small amount of
object sharding (S2) gives the best performance for reading data, and good
performance for writing until we scale up to the largest number of client
nodes. Full sharding gives the best write performance for high contention
scenarios (i.e. using the most client nodes) but lower performance for fewer
writers. We can also see that using the DFS API (i.e. the DAOS lines) gives
very similar performance to MPI-I/O using the DFuse mount, whereas HDF5 using
the DFuse mount gives much lower performance, both for read and write.

(a) Read

(b) Write

Figure 2: IOR: Shared-file

The benchmarking done with a single shared file shows different performance
characteristics, with similar performance achieved across interfaces. The DFS
API gives the highest write bandwidth, and good read bandwidth, but both
MPI-I/O and HDF5 using the DFuse mount also provide good read performance.
Interesting, file-per-process and shared-file give similar overall
performance, which is in stark contrast to the performance standard parallel
filesystems provide.

##  V Conclusions

We have presented some initial results from our current research to benchmark
different access mechanisms and configurations for using DAOS from
applications. We have demonstrated that both the object choice, and the
interface/API used, can impact performance, but that exploiting DAOS via the
available file APIs can still provide good performance, at least for bulk I/O,
on a DAOS object store.

Future work will include extending benchmarking to use the DAOS API (rather
than DFS or DFuse POSIX-based backends), and looking at some application
specific I/O benchmarks to evaluate the kind of performance more varied usage
patterns will experience.

## Acknowledgment

The work presented in this paper was carried out with funding by the European
Union under the Destination Earth initiative (cost center DE3100, code 3320)
and relates to tasks entrusted by the European Union to the European Centre
for Medium-Range Weather Forecasts. Views and opinions expressed are those of
the author(s) only and do not necessarily reflect those of the European Union
or the European Commission. Neither the European Union nor the European
Commission can be held responsible for them.

The NEXTGenIO system was funded by the European Union’s Horizon 2020 Research
and Innovation program under Grant Agreement no. 671951, and supported by
EPCC, The University of Edinburgh.

## References

  * [1] G. Lockwood, ”What’s so bad about POSIX I/O?”. The Next Platform 2017. https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/
  * [2] Z. Liang, J. Lombardi, M. Chaarawi, M. Hennecke, ”DAOS: A Scale-Out High Performance Storage Stack for Storage Class Memory” In: Panda, D. (eds) Supercomputing Frontiers. SCFA 2020. Lecture Notes in Computer Science(), vol 12082. Springer, Cham. DOI:10.1007/978-3-030-48842-0_3.
  * [3] J. Kunkel, J. Ben, J. Lofstead, G. Markomanolis, ”Establishing the IO-500 Benchmark”, White Paper, 2016
  * [4] K. Jeong, C. Duffy, J. -S. Kim and J. Lee, ”Optimizing the Ceph Distributed File System for High Performance Computing” 2019 27th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), 2019, pp. 446-451, DOI:10.1109/EMPDP.2019.8671563.
  * [5] S. Narasimhamurthy, N. Danilov, S. Wu, G. Umanesan, S. Chien, S. Rivas-Gomez, I. Peng, E. Laure, S. Witt, D. Pleiter, and S. Markidis. ”The SAGE project: a storage centric approach for exascale computing: invited paper”. In Proceedings of the 15th ACM International Conference on Computing Frontiers (CF ’18). Association for Computing Machinery, New York, NY, USA, 287–292. DOI:10.1145/3203217.3205341.
  * [6] A. Dilger, D. Hildebrand, J. Kunkel, J. Lofstead, and G. Markomanolis, ”IO500 10 node list Supercomputing 2022”, November 2022. https://io500.org/list/sc22/ten?sort=io500_md&direction=desc
  * [7] S. Smart, T. Quintino, and B. Raoult ”A Scalable Object Store for Meteorological and Climate Data”. In Proceedings of the Platform for Advanced Scientific Computing Conference (PASC ’17). Association for Computing Machinery, New York, NY, USA, Article 13, 1–8. DOI:10.1145/3093172.3093238.
  * [8] S. Smart, T. Quintino, and B. Raoult. ”A High-Performance Distributed Object-Store for Exascale Numerical Weather Prediction and Climate”. In Proceedings of the Platform for Advanced Scientific Computing Conference (PASC ’19). Association for Computing Machinery, New York, NY, USA, Article 16, 1–11. DOI:10.1145/3324989.3325726.
  * [9] Vef, MA., Moti, N., Süß, T. et al. ”GekkoFS — A Temporary Burst Buffer File System for HPC Applications”. J. Comput. Sci. Technol. 35, 72–91 (2020). DOI:10.1007/s11390-020-9797-6.
  * [10] O. Tatebe, K. Obata, K. Hiraga, and H. Ohtsuji. 2022. ”CHFS: Parallel Consistent Hashing File System for Node-local Persistent Memory”. In International Conference on High Performance Computing in Asia-Pacific Region (HPCAsia2022). Association for Computing Machinery, New York, NY, USA, 115–124. DOI:10.1145/3492805.3492807.
  * [11] A. Miranda, A. Jackson, T. Tocci, I. Panourgias and R. Nou, ”NORNS: Extending Slurm to Support Data-Driven Workflows through Asynchronous Data Staging” 2019 IEEE International Conference on Cluster Computing (CLUSTER), pp. 1-12, DOI:10.1109/CLUSTER.2019.8891014.
  * [12] N. Manubens, S. Smart, T. Quintino and A. Jackson, ”Performance Comparison of DAOS and Lustre for Object Data Storage Approaches”, 2022 IEEE/ACM International Parallel Data Systems Workshop (PDSW). DOI:10.1109/PDSW56643.2022.00007.
  * [13] ”DAOS Architecture”, 2023.
https://docs.daos.io/latest/overview/architecture

  * [14] ”HPC IO Benchmark Repository”, 2022, GitHub repository.
https://github.com/hpc/ior

  * [15] N. Manubens, A. Jackson, T. Quintino, S. Smart and E. Danovaro, ”DAOS weather field I/O tests”, 2023, GitHub repository ecmwf-projects/daos-tests (0.1.1). DOI:10.5281/zenodo.7559332.
  * [16] ”NEXTGenIO User Guide and Applications”, 2023.
https://ngioproject.github.io/nextgenio-docs/html/index.html

  * [17] ”benchio”,2023,https://github.com/adrianjhpc/benchio
  * [18] Weil, S., Brandt, S., Miller, E., Long, D. & Maltzahn, C. Ceph: A Scalable, High-Performance Distributed File System. 7th USENIX Symposium On Operating Systems Design And Implementation (OSDI 06). (2006,11), https://www.usenix.org/conference/osdi-06/ceph-scalable-high-performance-distributed-file-system
  * [19] https://www.ornl.gov/news/frontier-supercomputer-debuts-worlds-fastest-breaking-exascale-barrier
  * [20] N. Manubens, T. Quintino, S. D. Smart, E. Danovaro and A. Jackson, ”DAOS as HPC Storage: a View From Numerical Weather Prediction,” 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS), St. Petersburg, FL, USA, 2023, pp. 1029-1040, doi: 10.1109/IPDPS54959.2023.00106.
  * [21] Kordenbrock, Todd, Templet, Gary, Ulmer, Craig, and Widener, Patrick. Viability of S3 Object Storage for the ASC Program at Sandia. United States: N. p., 2022. Web. doi:10.2172/1895203.