# TS4Net: Two-Stage Sample Selective Strategy for Rotating Object Detection

Kai Feng, Weixing Li, Jun Han, Feng Pan, Dongdong Zheng
Beijing Institute of Technology, Beijing, China
<EMAIL_ADDRESS>
###### Abstract

Rotating object detection has wide applications in aerial photographs, remote
sensing images, UAVs, etc. At present, most of the rotating object detection
datasets focus on the field of remote sensing, and these images are usually
shot in high-altitude scenes. However, image datasets captured at low-altitude
areas also should be concerned, such as drone-based datasets. So we present a
low-altitude drone-based dataset, named UAV-ROD, aiming to promote the
research and development in rotating object detection and UAV applications.
The UAV-ROD consists of 1577 images and 30,090 instances of car category
annotated by oriented bounding boxes. In particular, The UAV-ROD can be
utilized for the rotating object detection, vehicle orientation recognition
and object counting tasks. Compared with horizontal object detection, the
regression stage of the rotation detection is a tricky problem. In this paper,
we propose a rotating object detector TS4Net, which contains anchor refinement
module (ARM) and two-stage sample selective strategy (TS4). The ARM can
convert preseted horizontal anchors into high-quality rotated anchors through
two-stage anchor refinement. The TS4 module utilizes different constrained
sample selective strategies to allocate positive and negative samples, which
is adaptive to the regression task in different stages. Benefiting from the
ARM and TS4, the TS4Net can achieve superior performance for rotating object
detection solely with one preseted horizontal anchor. Extensive experimental
results on UAV-ROD dataset and three remote sensing datasets DOTA, HRSC2016
and UCAS-AOD demonstrate that our method achieves competitive performance
against most state-of-the-art methods.

##  1 Introduction

Figure 1: Using drones to achieve rotating object detection and orientation
recognition. We propose TS4Net to localize the car captured by drone, and
utilize the ARM and TS4 module to improve the detection accuracy.

Object detection is one of the most fundamental and challenging tasks in
computer vision. In recent years, with the development of deep convolutional
neural networks (CNN), many high-performance object detectors have been
proposed [28, 17, 26, 27, 15, 31]. In general, the current popular detectors
can be divided into two categories: two-stage detectors and one-stage
detectors. The accuracy of two-stage methods is higher, while the speed of
one-stage methods is faster. These mainstream methods adopt horizontal
bounding boxes to locate the position of the object detection results.
However, horizontal detection suffers from many application limitations in
some fields, such as drone-based object detection and remote sensing object
detection. The intervention of the object angle also creates difficulties for
the post-processing of the detector, such as non-maximum suppression (NMS).

Rotating object detection aims to identify the category of a specific object
in image (e.g., car, ship, plane, etc.), and utilizes a rotation box to locate
the object (see Figure 1). Many researchers have proposed different rotating
object detection methods based on R-CNN frameworks [5, 39, 33, 36] or one-
stage method frameworks [34, 7, 24]. Compared with horizontal detection,
rotation detection seems more difficult to select positive samples from
horizontal anchors, which should have large Intersection-over-Union (IoU) with
the ground-truth. To fit with IoU matching, some literatures [18, 12, 19]
adopt rotated anchors as the original training samples. Utilizing the rotated
anchor can improve the location accuracy, but it also inevitably increases the
computational complexity, especially for high-resolution aerial images. How to
preset a few anchors or even one anchor to achieve high performance thus
becomes a major research spot in rotating object detection. During training,
the horizontal detection method adopts the constant IoU values to divide the
positive and negative samples. But small angle changes in rotation detection
can greatly affect the IoU, and thus the predefined IoU thresholds are not
conducive to dividing the samples. So a suitable sample selective strategy is
critical to reduce the variability between horizontal anchors and rotated
ground-truths.

To solve the above problems, we propose the TS4Net rotating object detection
method based on RetinaNet. TS4Net contains two modules: anchor refinement
module (ARM) and two-stage sample selective strategy (TS4). The ARM adopts two
serial networks to refine the preseted anchors. In the first stage, the ARM
adjusts the preseted horizontal anchors into high-quality rotated anchors
through the fully convolutional sub-networks. In the second stage, the input
rotated anchors are performed second refinement. In two stage regression task
of the ARM, the anchor refinement strategies for different stages should be
specifically considered. In the first stage, the gap between the preseted
horizontal anchor and the ground-truth is too large, thus we only need to take
a less restrictive sample selective strategy into consideration. However, the
spatial location similarity between the refined anchor and ground-truth is
high after the first stage of regression, so the more restrictive strategy
should be adopted in the second stage. The TS4 module makes the sample
distribution more reasonable by using different degrees of restrictive sample
selective methods, and can make the training more effective.

Considering existing rotating object detection datasets, most of the data are
remote sensing images with high-altitude view, such as DOTA [32], HRSC2016
[20] and UCAS-AOD [43]. These datasets have greatly promoted the development
of remote sensing object detection. However, there is a shortage of relevant
datasets in the field of aerial photographs from drones. In the drone-based
object detection, the drone can rotate at arbitrary-orient, so the drone-based
rotating object detection also has important practical application value.
Current object detection datasets for drones are usually annotated by
horizontal bounding boxes, such as UAVDT [6] and VisDrone2018 [44]. Therefore,
in order to enrich the data of drone-based rotating object detection, we
present a dataset named UAV-ROD. The dataset contains 1577 aerial images and
30,090 car instances annotated by oriented bounding boxes. In the annotating
process, we adopt the head orientation of car as the angle direction, so the
orientation recognition task can also be realized in UAV-ROD. The dataset
analysis is detailed in Section 3.

Our main contributions are summarized as follows:

  * •

We propose a simple and efficient rotating object detector TS4Net. Compared
with other dense detectors, TS4Net just needs to preset one anchor to achieve
superior results.

  * •

For the regression problem from the horizontal anchor to the rotated ground-
truth, we propose a two-stage sample selective strategy (TS4) with two-stage
anchor refinement module (ARM). Benefiting from ARM and TS4 module, the
accuracy of TS4Net has greatly improved compared with baseline.

  * •

We present a rotating object detection dataset, named UAV-ROD. Compared with
other drone-based datasets, UAV-ROD can be used for a variety of visual tasks,
such as rotating object detection, vehicle orientation recognition and object
counting111The images and annotations of UAV-ROD is available at
https://github.com/fengkaibit/UAV-ROD.

  * •

We achieve a competitive performance against most state-of-the-art methods on
UAV-ROD dataset and three remote sensing datasets DOTA, HRSC2016 and UCAS-AOD.

##  2 Related Work

###  2.1 Aerial Image Datasets

In recent years, many high-quality aerial image datasets have been proposed
for computer vision tasks. Hsieh et al. [9] proposed a dataset named CARPK
collected by drones in different parking lots, which contains 1448 images and
89,777 annotated cars. Zhu et al. [44] proposed a drone-based benchmark, named
VisDrone2018, consisting of 10,209 images and 263 videos with horizontal
bounding box annotations. Xia et al. [32] proposed the DOTA dataset, which
contains 2806 remote sensing images, and the average size of image is about
4000pixels. The DOTA dataset contains 15 different categories and about
400,000 instances, annotated by horizontal bounding boxes and oriented
bounding boxes, respectively. In [20], a dataset named HRSC2016 for ship
detection was proposed, which includes 1061 aerial images and about 3000
annotated ships. Zhu et al. [43] proposed the UCAS-AOD dataset, consisting of
1510 remote sensing images and two categories including plane and car. Cheng
et al. [4] proposed a multi-class geospatial dataset, named NWPU VHR-10. It
consists of 800 images and 10 categories, and a total of 3651 instances are
annotated by horizontal bounding boxes. Long et al. [22] proposed the RSOD
dataset, which contains 976 remote sensing images and 4 categories. Du et al.
[6] proposed a large scale vehicle detection and tracking dataset captured by
drones, named UAVDT, which contains about 80,000 representative video frames,
10 hours of original video and 14 attributes of vehicles, such as weather
conditions, flight height, vehicle category, horizontal bounding box and
occlusion.

In summary, existing drone-based datasets are deficient in rotating
annotation, so the UAV-ROD dataset focuses on drone-based detection and
rotation detection tasks when collecting data. We proposed the UAV-ROD dataset
as a benchmark to help evaluate the performance of rotating object detection
algorithms. The detailed comparison of the UAV-ROD with other datasets is
presented in Table 1.

###  2.2 Object Detection in Aerial Images

Remote sensing and aerial photography are the main application scenarios of
the rotation detection. In these scenarios, the objects are usually densely
distributed, with a large aspect ratio and arbitrary angle. The current
mainstream rotation detection methods can be divided into two categories like
horizontal object detection: two-stage detectors and one-stage detectors. The
two-stage detectors consist of two parts: a region proposal network (RPN) and
R-CNN network. The RPN network uses the preseted horizontal anchors to
generate high-quality region of interests (RoIs), and then the R-CNN network
continues to classify and regress them. Due to the multi-stage processing and
the RoI Pooling methods for feature cropping, the two-stage methods still
occupy a leading position in terms of accuracy. Representative detectors are
ROI-Transformer [5], ICN [1], SCRDet [36], R2CNN [10], etc. However, these
model structures are more complex, resulting in slow inference speed. The one-
stage methods achieve classification and regression through fully
convolutional networks, which is advantageous in terms of model structure and
inference speed. But their accuracy still trails that of two-stage methods.
Focal Loss [15] lowers the weight of easy samples to avoid imbalance problems.
So the one-stage method can also match the accuracy of the two-stage method
after integrating the Focal Loss. Therefore, many one-stage rotating object
detection algorithms have been proposed, such as R3Det [34] and S2ANet [7].
The above methods achieve state-of-the-art accuracy, but ignore analyzing the
anchor selective strategies in different training stages. So the selected
candidate samples seem to be unsatisfying, which are not favorable for model
training.

###  2.3 Label Assignment

Most detectors select positive and negative samples through a specific
strategy, which is called label assignment. The commonly used label assignment
method is to divide the positive and negative samples by the IoU between the
anchor box and the ground-truth, which is called the Max-IoU method. For
instance, RetinaNet selects positive samples with IoU values higher than 0.5,
and those whose IoU values lower than 0.4 are considered to be negative
samples. Dynamic R-CNN [40] adjusts the IoU thresholds at different stages to
select high-quality positive samples dynamically. Cascade R-CNN [3] sets
different IoU thresholds for different cascade detection heads to select high-
quality positive samples. The Max-IoU method selects the samples by measuring
the spatial relationship between the preseted anchor and the ground-truth, and
has strong spatial constraints to the candidate samples. In rotating object
detection, the ground-truth has an arbitrary angle, and the angle change will
greatly affect the IoU. Therefore, the Max-IoU method is not conducive to
selecting a large number of high-quality candidate samples in rotation
detection task.

In recent years, with the emergence of the anchor-free method, the label
assignment strategy based on central distance relationship has been proposed.
FCOS [29] divides positive and negative samples by measuring the spatial-scale
limitation and whether the anchor point falls into the ground-truth or not.
ATSS [41] analyzes the main differences between FCOS and RetinaNet and
concludes the sample selective strategy is the main reason for the difference
of the accuracy. In this way, a label assignment strategy based on the center
distance constraints is proposed. ATSS selects the candidate samples through
the center distance relationship and determines the positive threshold from
the statistical information. The label assignment strategy based on the
distance constraints measures the importance of the sample through the center
distance information, and can obtain a large number of candidate positive
samples. However, in rotating object detection, this label assignment strategy
does not consider the angle information, so it is a less restrictive sample
selective strategy, which is not conducive to further assigning high quality
samples.

##  3 UAV-ROD Dataset

###  3.1 Image Collection

Compared with remote sensing and high-altitude aerial photography datasets, we
use drone to collect images in low-altitude. The flying height of the drone is
between 30 meters and 80 meters. The image scenes include city roads, parking
lots, residential areas, roadsides, etc. UAV-ROD dataset contains 10 aerial
videos in different scenarios, and we compose the dataset by extracting images
at 30 frame intervals. Finally, the UAV-ROD dataset has 1577 images whose
resolution is 1920.

Table 1: Comparison of aerial image datasets. Oriented BB stands for oriented bounding box. Dataset | Scenario |  $\\#$Images |  $\\#$Categories |  $\\#$Instances | Image width | Oriented BB
---|---|---|---|---|---|---
DOTA [32] | aerial | 2,806 | 15 | 188.3 _k_ | 800$\sim$4000 | $\checkmark$
HRSC2016 [20] | aerial | 1,061 | 1 | 2.9 _k_ |  $\sim$1100 | $\checkmark$
UCAS-AOD [43] | aerial | 1,510 | 2 | 14.6 _k_ |  $\sim$1000 | $\checkmark$
NWPU VHR-10 [4] | aerial | 800 | 10 | 3.7 _k_ |  $\sim$1000 |
RSOD [22] | aerial | 976 | 4 | 6.9 _k_ | 800$\sim$1100 |
UAVDT [6] | drone | 80,000 | 3 | 841.5 _k_ |  $\sim$1100 |
VisDrone [44] | drone | 10,209 | 10 | 542 _k_ |  $\sim$1920 |
CARPK [9] | drone | 1,448 | 1 | 89.8 _k_ | 1280 |
UAV-ROD (ours) | drone | 1,577 | 1 | 30.0 _k_ |  $\sim$1920 | $\checkmark$

###  3.2 Data Annotation

In the field of visual object detection, the typical method of annotation
mainly adopts horizontal bounding boxes to locate the object position on
image. The horizontal bounding box is marked with ($x_{c}$, $y_{c}$, $w$,
$h$), where ($x_{c}$, $y_{c}$) is the center of the object, and $w$, $h$ are
the width and height of the box. Some methods are also annotated with
($x_{min}$, $y_{min}$, $x_{max}$, $y_{max}$) format, where ($x_{min}$,
$y_{min}$) and ($x_{max}$, $y_{max}$) are the top-left and the bottom-right
corner point of the object, respectively. Horizontal annotations are widely
used in natural scene images. However, in aerial object detection scenes,
using horizontal annotation may cause inaccuracies in object position.
Especially when the object has an arbitrary angle, the horizontal box does not
fit the contour of the object closely. So we adpot the oriented bounding box
for annotation.

Figure 2: Schematic diagram of the size and angle statistics of UAV-ROD
dataset. Figure 3: Some example annotated images of the UAV-ROD dataset.

There are two main annotation methods used for rotating object annotation:
four-point annotation method and oriented bounding box method. The four-point
annotation method is used in datasets such as DOTA [32] and UCAS-AOD [43]. In
detail, the annotation form of this method is {($x_{i}$, $y_{i}$),
$i$=1,2,3,4}, where ($x_{i}$, $y_{i}$) denotes the vertex position of the
object. The $\theta$-based oriented bounding box is adopted in HRSC2016 [20]
and some text detection benchmark [37], and its representation form is
($x_{c}$, $y_{c}$, $w$, $h$, $\theta$). In UAV-ROD, $\theta$ denotes the angle
of car head direction and is increased in a clockwise manner. The $w$, $h$
represent the width and height of the oriented box respectively. Figure 2
shows the statistics of the annotated size and angle of the instances.

###  3.3 Data Statistics

The UAV-ROD dataset consists of 1577 images captured by drone, and 30,090 car
instances annotated with the oriented bounding box method. The average number
of objects per image in our dataset is 19.08, of which the maximum number of
objects per image is 104. In the image with dense objects, we also annotate
all the instances in the dense area one by one. In contrast, dense objects are
usually not annotated one-by-one in natural scene datasets due to the
occlusion. The UAV-ROD dataset can be used in rotating object detection,
vehicle orientation recognition and object counting tasks. Some examples of
annotated images in our dataset are shown in Figure 3.

##  4 Method

###  4.1 RetinaNet as Baseline

We adopt the vanilla one-stage detector, RetinaNet [15] as our baseline. It
uses ResNet [8] as the backbone, and constructs mutil-level features through
feature pyramid networks (FPN) [14]. Two parallel fully convolutional networks
are connected after FPN to carry out the classification and regression tasks
respectively. In addition, the Focal Loss is applied to overcome the imbalance
problem during training. The official RetinaNet is designed for horizontal
object detection, we incorporate the $\theta$ parameter in the oriented box
regression and have the oriented box prediction as ($x_{c}$, $y_{c}$, $w$,
$h$, $\theta$). We follow the definition in [5] and adopt
$\theta\in[-\pi/4,3\pi/4)$. The $w,h$ are the long and short side of the
oriented box respectively. An example is shown in Figure 4, where $\theta$
means the angle from the x-axis to the direction of $w$. For regression task,
we have:

$\begin{split}t_{x}=(x-x_{a})/w_{a},\quad t_{y}=(y-y_{a})/h_{a}\\\
t_{w}=log(w/w_{a}),\quad t_{h}=log(h/h_{a})\quad\\\
t_{\theta}=\theta-\theta_{a}+k\pi\qquad\qquad\end{split}$ (1)

$\begin{split}t^{{}^{\prime}}_{x}=(x^{{}^{\prime}}-x_{a})/w_{a},\quad
t^{{}^{\prime}}_{y}=(y^{{}^{\prime}}-y_{a})/h_{a}\\\
t^{{}^{\prime}}_{w}=log(w^{{}^{\prime}}/w_{a}),\quad
t^{{}^{\prime}}_{h}=log(h^{{}^{\prime}}/h_{a})\quad\\\
t^{{}^{\prime}}_{\theta}=\theta^{{}^{\prime}}-\theta_{a}+k\pi\qquad\qquad\end{split}$
(2)

where $(x,y),w,h,\theta$ denote the bounding box center point, width, height
and angle, respectively. $x$, $x_{a}$ and $x^{{}^{\prime}}$ are for the
ground-truth box, anchor and predicted box, respectively (likewise for
$y,w,h,\theta$). $k$ is an integer to ensure
($\theta-\theta_{a})\in[-\pi/4,3\pi/4)$. Given the ground-truth box offsets
$\bm{t}=(t_{x},t_{y},t_{w},t_{h},t_{\theta})$ and the predicted offsets
$\bm{t^{{}^{\prime}}}=(t^{{}^{\prime}}_{x},t^{{}^{\prime}}_{y},t^{{}^{\prime}}_{w},t^{{}^{\prime}}_{h},t^{{}^{\prime}}_{\theta})$,
the regression loss is calculated between $\bm{t}$ and $\bm{t^{{}^{\prime}}}$.
For testing, the predicted offsets $\bm{t^{{}^{\prime}}}$ is inversely solved
according to Equation 2 to obtain the predicted box.

Figure 4: Oriented bounding box definition in TS4Net. $c$ denotes the center
point. $\theta$, $w$ and $h$ represent the angle, long side and the short side
of bounding boxes respectively.

###  4.2 Anchor Refinement Module

Compared with horizontal detection, rotation detection is more difficult to
regress the preseted horizontal anchors to rotated ground-truths during
training. In particular, it is a challenge to select the positive samples from
the horizontal anchors with large IoU values. To address this issue, we adopt
a two-stage anchor refinement module (ARM) in TS4Net (see Figure 5). The ARM
includes a two-stage cascade network, and each network contains parallel
classification and regression sub-networks, which are used for category
prediction and rotated box regression respectively. The ARM first classifies
and regresses the preseted anchor in the first stage and transforms the
horizontal anchor into high-quality rotated anchor, facilitating the second
stage of the anchor regression process. In the second stage, the rotated
anchors are fed into the second subnetwork to yield more accurate prediction
results. Profiting from the ARM, the single-stage detector can be optimized
several times for the preseted anchor like the multi-stage detector, thus
making the detection results more precise. In order to improve the inference
speed, the two stages of the ARM use two 256-channel convolutional layers as
classification and regression networks. During training, the total loss
contains the two-stage loss of the ARM, as shown in Equation 3:

$\small\begin{split}L=\frac{1}{N^{1}_{cls}}\sum_{i}L^{1}_{cls}(p_{i},p^{*}_{i})+\lambda\frac{1}{N^{1}_{reg}}\sum_{i}p^{*}_{i}L^{1}_{reg}(t_{i},t^{*}_{i})\\\
+\frac{1}{N^{2}_{cls}}\sum_{i}L^{2}_{cls}(p_{i},p^{*}_{i})+\lambda\frac{1}{N^{2}_{reg}}\sum_{i}p^{*}_{i}L^{2}_{reg}(t_{i},t^{*}_{i})\end{split}$
(3)

where $L_{cls}$ and $L_{reg}$ are implemented by focal loss and smooth L1
loss, $N_{cls}$ and $N_{reg}$ are the total number of samples involved in
classification and regression, respectively. The superscripts 1 and 2
respectively represent the first and second stage of the ARM. Variable $p^{*}$
represents the class label for anchors ($p^{*}=1$ for positive samples and
$p^{*}=0$ for negative samples). The hyper-parameter $\lambda$ controls the
trade-off and is set to 1 by default.

Figure 5: Architecture of the proposed TS4Net. TS4Net consists of a ResNet
backbone, a FPN, an ARM cascade network and the two-stage sample selective
strategy. ‘C’ indicates number of categories.

###  4.3 Two-stage Sample Selective Strategy

In this section, we discuss the differences between two sample selective
strategies in rotation detection from different perspectives, and propose a
two-stage sample selective strategy to combine their advantages.

Sample selective strategy based on spatial information. The Max-IoU is a
typical method that uses spatial information to select samples. Specifically,
an anchor is classified as a positive sample when the IoU value between the
anchor and the ground-truth is greater than the positive threshold, and as a
negative sample when the IoU is lower than the negative threshold. This sample
selective strategy is widely used in object detection. The classical object
detector enhances the IoU value by setting a lot of anchors with different
sizes and aspect ratios, thus boosting the number of positive samples.
However, in the rotation detection, the angle leads to a low IoU value between
the horizontal anchor and the ground-truth. Moreover, the IoU value is greatly
affected by the rotation angle, which causes a poor quality of the obtained
positive samples. To alleviate this problem, many rotation detectors add
preseted angles to the anchor, thus improving the IoU value with ground-truth.
However, it will significantly increase the quantity of anchors, which will
greatly aggravate the computational efficiency of the model for high-
resolution aerial images.

Sample selective strategy based on central distance information. In FCOS and
ATSS algorithms, the significance of samples is measured by the distance
between the center point of the samples and the ground-truth. Compared with
the Max-IoU method, the method based on the center distance is less
constrained, and enables a larger number of positive samples to be obtained.
Moreover, this method sets only one sample at each anchor point, which
significantly reduces the computations and improves the efficiency. In the
rotating object detection, the ground-truth contains the angle information of
the object. However this strategy does not utilize the angle information in
it, which is not conducive to further selecting the high quality candidate
positive samples. In contrast, the calculation of IoU includes the
relationship of angle information between the sample and the ground-truth,
which is a strongly constrained strategy. Therefore, combining the advantages
of the two strategies is our main motivation for proposing the TS4 module.

Two-stage sample selective strategy. From the previous analysis, TS4Net
incorporates ARM to obtain more accurate localization results. The first stage
of ARM refines the horizontal anchors to high quality rotated anchors, and
then the second stage adjusts the rotating anchor to a more accurate
prediction box. Both stages require to use the sample selective strategy. In
the first stage, the detector needs to adjust the horizontal anchors to the
rotated anchors. Due to the large gap between the horizontal anchor and its
corresponding ground-truth, thus the less constrained ATSS sample selective
strategy is adopted in first stage. The ATSS strategy divides positive and
negative samples according to the statistical information of the central
distance between samples and ground-truths, and obtains high quality
horizontal anchors as positive samples with a moderately easy sample
constraint. After the first stage of classification and regression, the
horizontal anchors are refined to high quality rotated anchors. At this time,
we achieve a high value of IoU and spatial similarity between the rotated
anchor and ground-truth. Therefore, the model adopts Max-IoU method with
strict constraint as the sample selective strategy in second stage, and
obtains more accurate positive samples. The network architecture of TS4Net is
shown in Figure 5. The first stage of TS4Net utilizes the horizontal anchors
and ATSS strategy to complete the sample selection, and the second stage uses
the rotated anchors and Max-IoU method instead. Both stages contain two
parallel sub-networks for classification and regression, similar to RetinaNet.

##  5 Experiments

###  5.1 Datasets

We conduct experiments on the UAV-ROD dataset and the remote sensing datasets
DOTA, HRSC2016, UCAS-AOD. The ground-truth of these datasets are all oriented
bounding boxes. UAV-ROD is a drone-based car detection dataset, which contains
1150 images in the training set and 427 images in the test set. DOTA is a
large remote sensing dataset, including 2806 images and 188,282 annotated
instances of 15 categories. Half of the images are randomly assigned as the
training set, 1/6 as the validation set and 1/3 as the test set. Since the
image resolution of DOTA is too large, we crop the original images into
1024patches with the stride of 200. HRSC2016 is a high-resolution ship
recognition dataset, which contains 1061 images. The training, validation and
test set on HRSC2016 include 436 images, 181 images and 444 images,
respectively. UCAS-AOD is a plane and car detection dataset containing a total
of 1510 images. Following the settings in the literature [32, 1], we randomly
select 1110 images for training and 400 images for testing. During training,
the input image size of DOTA is 1024, and other dataset images are rescaled to
1333.

###  5.2 Implementation Details

For the experiments, we adopt RetinaNet as baseline according to the above
description. ResNet-50 is adopted as backbone and FPN is used to build the
5-level feature layers (i.e., $P_{3}$ \- $P_{7}$). TS4Net sets one square
anchor box per location, with sizes {32, 64, 128, 256, 512} in $P_{3}$ \-
$P_{7}$. The total epochs of UAV-ROD, DOTA, HRSC2016 and UCAS-AOD are 12, 12,
72 and 24, respectively. We use random horizontal flipping and rotation for
data augmentation. The optimizer used for training is SGD with an initial
learning rate of 2.5e-3, which is divided by 10 at each decay step. The
momentum and weight decay are 0.9 and 0.0001, respectively. We train the model
on two V100 GPUs with a total batch size of 4.

###  5.3 Ablation Study

Table 2: Ablation study for TS4Net on HRSC2016 dataset. We explore the effect of ARM, the number of anchors and the different forms of the two-stage sample selective strategy on accuracy. Method | Assigner |  $\\#$Anchor | ARM | mAP
---|---|---|---|---
RetinaNet | Max-IoU | 27 |  | 83.32
Max-IoU | 1 |  | 82.03
Max-IoU + Max-IoU | 1 | $\checkmark$ | 87.21
TS4Net | Max-IoU + Max-IoU | 1 | $\checkmark$ | 87.37
ATSS + ATSS | 1 | $\checkmark$ | 88.12
Max-IoU + ATSS | 1 | $\checkmark$ | 86.18
ATSS + Max-IoU | 1 | $\checkmark$ | 89.44

In this section, we construct a series of experiments on HRSC2016 to verify
the effectiveness of our method. All experiments adopt ResNet-50 FPN as the
backbone and use RetinaNet as the baseline.

RetinaNet Results. To achieve rotating object detection, we integrate the
angle prediction features described in subsection 4.1 into RetinaNet. And we
add three angles $[0,\pi/6,\pi/3]$ at each anchor to enhance the matching
effect of Max-IoU method. As shown in Table 2, RetinaNet achieves a mAP of
83.32% on HRSC2016 and is used for our baseline. We change the number of
anchor to 1, and obtain 82.03$\%$ of mAP. The result shows that few anchors
also has the potential to achieve high accuracy while can reduce the
computation cost of the model.

Anchor Refinement Module. To demonstrate the effectiveness of the ARM, we
replace the original detection head with the ARM head (see the fourth row of
Table 2). In this case, the sample selective strategy for both stages adopts
Max-IoU method and the positive and negative sample thresholds are set to 0.5
and 0.4, and we obtain 87.21% of mAP. Compared to the baseline and the
previous experiment (Table 2, row 3), the precision is improved by 3.89%
(87.21% $v.s.$ 83.32%) and 5.18% (87.21% $v.s.$ 82.03%) respectively. The
result demonstrates that with the two-stage anchor refinement, the model can
achieve high performance with only one preseted anchor.

Two-stage sample selective strategy. To demonstrate the correctness of the
analysis in section 4.3, we design four experiments to validate the
effectiveness of two-stage sample selective strategy. Firstly, we adopt Max-
IoU in both stages, and the positive and negative sample thresholds in the
second stage are increased to 0.6 and 0.5. The aim is to improve the quality
of the sample. We obtain a mAP of 87.37%. Changing the sample selective
strategy to ATSS in both stages, the mAP is 88.12%, which is 0.75% higher
compared to Max-IoU. Considering that the first stage is to refine horizontal
anchor to rotated anchor and the second stage is the fine-tune regression of
rotated anchor, we adopt the ATSS and Max-IoU methods in two sample selection
processes respectively. The mAP is 89.44%, which is higher than the previous
strategies. The experimental results verify our analysis. As a contrast
experiment, we test the combination of Max-IoU as the first stage and ATSS as
the second stage. We obtain a mAP of 86.18%, which is 3.26% lower compared to
the ATSS+Max-IoU strategy. This indicates that using a strongly constrained in
the first stage and a weakly constrained strategy in the second stage
contradicts the process of rotating box regression, thus does not yield a
credible result. The visualization of different strategies can be found in the
appendix.

###  5.4 Experiment Results

In this section, we compare our proposed TS4Net with other state-of-the-art
methods on UAV-ROD, DOTA, HRSC2016 and UCAS-AOD datasets.

Results on UAV-ROD. On UAV-ROD dataset, our method is compared with two
classic one-stage and two-stage detectors. Following the MS COCO evaluation
metrics, we report the AP, AP75 and AP50 results. As shown in Table 3, we
achieve the AP of 75.57% with ResNet-50, which outperforms the baseline model
by 4.11% (75.57% $v.s.$ 71.46%). The results demonstrated that with ARM and
TS4, our method is able to achieve better accuracy in localization. Compared
with a typical two-stage method R-Faster R-CNN, our method also achieves a
comparable accuracy (75.57% $v.s.$ 75.79%). Using ResNet-101 as backbone,
TS4Net obtains 76.42% AP and 98.34% AP50, achieving the best results.

Results of orientation recognition task on UAV-ROD. We adopt the UAV-ROD
dataset to perform the car orientation recognition task. We modify two
settings on TS4Net: (1) The range of angle prediction is adjusted to
$[0,2\pi)$, thus can cover all head directions of the car. (2) To avoid the
large offest of angle regression, the decode of angle is divided by 2$\pi$ in
Equation 1 and Equation 2. The predicted results of car orientation are shown
in Figure 6, where the red line indicates the head direction.

Table 3: Detection results on UAV-ROD dataset. We report the performance in MS COCO style. Method | Backbone | AP |  AP75 |  AP50
---|---|---|---|---
R-RetinaNet [15] | ResNet-50 | 71.46 | 85.88 | 97.68
R-Faster R-CNN [28] | ResNet-50 | 75.79 | 86.38 | 98.07
_ours:_ |  |  |  |
TS4Net | ResNet-50 | 75.57 | 86.70 | 98.03
TS4Net | ResNet-101 | 76.42 | 88.39 | 98.34

Results on DOTA. We compare our method with the state-of-the-arts on DOTA as
shown in Table 5. Our method achieves mAP of 74.70% with ResNet-50. Using
ResNet-101 as the backbone, our method has reached the mAP of 75.63% and
outperforms other advanced rotation detectors. More visualization of detection
results on DOTA can be found in the appendix.

Table 4: Detection results on HRSC2016 dataset. mAP* stands for evaluated by PASCAL VOC2012 metrics. Method | Backbone | #Anchor | mAP
---|---|---|---
R2CNN [10] | ResNet-101 | 21 | 73.07
RC1&RC2 [21] | VGG16 | - | 75.70
RRPN [23] | ResNet-101 | 54 | 79.08
R2PN [42] | VGG16 | 24 | 79.60
R-RetinaNet [15] | ResNet-50 | 27 | 83.32
RRD [13] | VGG16 | 13 | 84.3
RoI-Trans. [5] | ResNet-101 | 20 | 86.2
R3Det [34] | ResNet-101 | 21 | 89.26
DRN [25] | Hourglass104 | - | 92.7*
CenterMap-Net [30] | ResNet-50 | 15 | 92.8*
_ours:_ |  |  |
TS4Net | ResNet-50 | 1 | 89.44/92.60*
TS4Net | ResNet-101 | 1 |  89.87/ 94.79*

Results on HRSC2016. HRSC2016 is a challenging dataset, which contains a lot
of rotated ships with large aspect ratios. The test results of HRSC2016 are
depicted in Table 4. TS4Net achieves mAP of 89.44% (VOC2007) and 92.60%
(VOC2012) with ResNet-50 as backbone. With ResNet-101 backbone, our model
achieves 89.87% and 94.79% mAP under VOC2007 and VOC2012 metrics,
respectively, which outperforms the other methods. The experimental results
verify the effectiveness of our method.

Figure 6: Visualization of car orientation recognition. The red line indicates the head direction of the car. Table 5: Detection accuracy of OBB task on DOTA dataset. R-50 stands for ResNet-50 (likewise for R-101 and R-152), and H-104 represents Hourglass-104. Method | Backbone | PL | BD | BR | GTF | SV | LV | SH | TC | BC | ST | SBF | RA | HA | SP | HC | mAP
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
_two-stage:_ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
FR-O [32] | R-101 | 79.09 | 69.12 | 17.17 | 63.49 | 34.20 | 37.16 | 36.20 | 89.19 | 69.60 | 58.96 | 49.40 | 52.52 | 46.69 | 44.80 | 46.30 | 52.93
R-DFPN [35] | R-101 | 80.92 | 65.82 | 33.77 | 58.94 | 55.77 | 50.94 | 54.78 | 90.33 | 66.34 | 68.66 | 48.73 | 51.76 | 55.10 | 51.32 | 35.88 | 57.94
R2CNN [10] | R-101 | 80.94 | 65.67 | 35.34 | 67.44 | 59.92 | 50.91 | 55.81 | 90.67 | 66.92 | 72.39 | 55.06 | 52.23 | 55.14 | 53.35 | 48.22 | 60.67
RRPN [23] | R-101 | 88.52 | 71.20 | 31.66 | 59.30 | 51.85 | 56.19 | 57.25 | 90.81 | 72.84 | 67.38 | 56.69 | 52.84 | 53.08 | 51.94 | 53.58 | 61.01
ICN [1] | R-101 | 81.36 | 74.30 | 47.70 | 70.32 | 64.89 | 67.82 | 69.98 | 90.76 | 79.06 | 78.20 | 53.64 | 62.90 | 67.02 | 64.17 | 50.23 | 68.16
RoI-Trans. [5] | R-101 | 88.64 | 78.52 | 43.44 | 75.92 | 68.81 | 73.68 | 83.59 | 90.74 | 77.27 | 81.46 | 58.39 | 53.54 | 62.83 | 58.93 | 47.67 | 69.56
SCRDet [36] | R-101 | 89.98 | 80.65 | 52.09 | 68.36 | 68.36 | 60.32 | 72.41 | 90.85 | 87.94 | 86.86 | 65.02 | 66.68 | 66.25 | 68.24 | 65.21 | 72.61
_one-stage:_ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
R-RetinaNet [15] | R-50 | 89.27 | 82.09 | 38.09 | 70.97 | 74.93 | 64.04 | 78.15 | 90.82 | 86.40 | 76.25 | 59.75 | 67.07 | 54.62 | 69.85 | 49.49 | 70.12
DRN [25] | H-104 | 88.91 | 80.22 | 43.52 | 63.35 | 73.48 | 70.69 | 84.94 | 90.14 | 83.85 | 84.11 | 50.12 | 58.41 | 67.62 | 68.60 | 52.50 | 70.70
R3Det [34] | R-101 | 89.54 | 81.99 | 48.46 | 62.52 | 70.48 | 74.29 | 77.54 | 90.80 | 81.39 | 83.54 | 61.97 | 59.82 | 65.44 | 67.46 | 60.05 | 71.69
BBAVector [38] | R-101 | 88.35 | 79.96 | 50.69 | 62.18 | 78.43 | 78.98 | 87.94 | 90.85 | 83.58 | 84.35 | 54.13 | 60.24 | 65.22 | 64.28 | 55.70 | 72.32
R3Det [34] | R-152 | 89.49 | 81.17 | 50.53 | 66.10 | 70.92 | 78.66 | 78.21 | 90.81 | 85.26 | 84.23 | 61.81 | 63.77 | 68.16 | 69.83 | 67.17 | 73.74
_ours:_ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
TS4Net | R-50 | 89.33 | 83.73 | 50.26 | 71.33 | 78.82 | 75.28 | 86.31 | 90.85 | 84.54 | 85.51 | 63.60 | 65.57 | 66.51 | 72.01 | 56.88 | 74.70
TS4Net | R-101 | 89.59 | 84.07 | 53.41 | 73.23 | 78.76 | 77.11 | 87.04 | 90.88 | 85.94 | 85.48 | 63.04 | 64.98 | 68.05 | 72.34 | 60.58 | 75.63

Results on UCAS-AOD. The UACS-AOD dataset contains plane and car objects with
rotated ground-truth. The experimental results on the UACS-AOD dataset are
shown in Table 9. Our model achieves 96.09% mAP results with ResNet-50, and
96.44% mAP with ResNet-101 as backbone. Compared to the baseline, the mAP is
improved by 2.34% (96.09% $v.s.$ 93.75%). Compared with other methods in Table
9, our model achieves the highest accuracy and the best results under the car
category.

Table 6: Detection results on UCAS-AOD dataset. Method | mAP | Plane | Car
---|---|---|---
R-DFPN [35] | 89.20 | 95.90 | 82.50
DRBox [16] | 89.95 | 94.90 | 85.00
R-RetinaNet [15] | 93.75 | 97.79 | 89.71
S2ARN [2] | 94.90 | 97.60 | 92.20
ICN [1] | 95.67 | - | -
FADet [11] | 95.71 | 98.69 | 92.72
R3Det [34] | 96.17 | 98.20 | 94.14
_ours:_ |  |  |
TS4Net-R50 | 96.09 | 98.58 | 93.60
TS4Net-R101 | 96.44 | 98.42 | 94.46

##  6 Conclusion

In this paper, we present a drone-based rotating object detection dataset
named UAV-ROD, and believe that our proposed dataset can promote the progress
of the rotating object detection community. Besides, we design a rotating
object detector TS4Net based on the two-stage sample selective strategy.
Combined with the anchor refinement module and the two-stage sample selective
strategy, the TS4Net can address the rotated regression problem of horizontal
anchor satisfactorily. Extensive experiments have demonstrated our method can
achieve competitive performance against most state-of-the-art methods.

##  7 Appendix

###  7.1 Analysis of different two-stage sample seletive strategies.

Figure 7 shows the positive sample visualization results for the two different
strategies. As can be seen from Figure 7, the Max-IoU method is more
constrained in selecting positive samples, and the number of high-quality
positive samples selected is low, making the training inadequate and
imbalance. The ATSS+Max-IoU method selects sufficient positive horizontal
samples by the center distance information in first-stage, and then selects
high-quality rotated anchors for the second regression by the constrained Max-
IoU method. This method can select sufficient high-quality samples, thus
benefiting the training process.

Figure 7: Visualization of different two-stage sample seletive strategy
results.

###  7.2 Qualitative Results

We show some qualitative detection results on different datasets, including
UAV-ROD (Figure 10), DOTA (Figure 11), HRSC2016 (Figure 8) and UCAS-AOD
(Figure 9) datasets.

Figure 8: Qualitative results of TS4Net on HRSC2016 dataset. Figure 9:
Qualitative results of TS4Net on UCAS-AOD dataset. Figure 10: Qualitative
results of TS4Net on UAV-ROD dataset. Figure 11: Qualitative results of TS4Net
on DOTA dataset.

## References

  * [1] Seyed Majid Azimi, Eleonora Vig, Reza Bahmanyar, Marco Körner, and Peter Reinartz.  Towards multi-class object detection in unconstrained remote sensing imagery.  In Proceedings of Asian Conference on Computer Vision, pages 150–165, 2018.
  * [2] Songze Bao, Xing Zhong, Ruifei Zhu, Xiaonan Zhang, Zhuqiang Li, and Mengyang Li.  Single shot anchor refinement network for oriented object detection in optical remote sensing imagery.  IEEE Access, 7:87150–87161, 2019.
  * [3] Zhaowei Cai and Nuno Vasconcelos.  Cascade R-CNN: delving into high quality object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6154–6162, 2018.
  * [4] Gong Cheng, Peicheng Zhou, and Junwei Han.  Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images.  IEEE Transactions on Geoscience and Remote Sensing, 54(12):7405–7415, 2016.
  * [5] Jian Ding, Nan Xue, Yang Long, Gui-Song Xia, and Qikai Lu.  Learning roi transformer for oriented object detection in aerial images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2849–2858, 2019.
  * [6] Dawei Du, Yuankai Qi, Hongyang Yu, Yi-Fan Yang, Kaiwen Duan, Guorong Li, Weigang Zhang, Qingming Huang, and Qi Tian.  The unmanned aerial vehicle benchmark: Object detection and tracking.  In Proceedings of the European Conference on Computer Vision, pages 375–391, 2018.
  * [7] Jiaming Han, Jian Ding, Jie Li, and Gui-Song Xia.  Align deep features for oriented object detection.  IEEE Transactions on Geoscience and Remote Sensing, pages 1–11, 2021.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [9] Meng-Ru Hsieh, Yen-Liang Lin, and Winston H. Hsu.  Drone-based object counting by spatially regularized regional proposal network.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4165–4173, 2017.
  * [10] Yingying Jiang, Xiangyu Zhu, Xiaobing Wang, Shuli Yang, Wei Li, Hua Wang, Pei Fu, and Zhenbo Luo.  R2CNN: rotational region CNN for orientation robust scene text detection.  arXiv preprint arXiv: 1706.09579, 2017.
  * [11] Chengzheng Li, Chunyan Xu, Zhen Cui, Dan Wang, Tong Zhang, and Jian Yang.  Feature-attentioned object detection in remote sensing imagery.  In Proceedings of the IEEE International Conference on Image Processing, pages 3886–3890, 2019.
  * [12] Minghui Liao, Baoguang Shi, and Xiang Bai.  Textboxes++: A single-shot oriented scene text detector.  IEEE Transactions on Image Processing, 27(8):3676–3690, 2018.
  * [13] Minghui Liao, Zhen Zhu, Baoguang Shi, Gui-Song Xia, and Xiang Bai.  Rotation-sensitive regression for oriented scene text detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5909–5918, 2018.
  * [14] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 936–944, 2017.
  * [15] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2999–3007, 2017.
  * [16] Lei Liu, Zongxu Pan, and Bin Lei.  Learning a rotation invariant detector with rotatable bounding box.  arXiv preprint arXiv: 1711.09405, 2017.
  * [17] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: single shot multibox detector.  In Proceedings of the European Conference on Computer Vision, pages 21–37, 2016.
  * [18] Wenchao Liu, Long Ma, and He Chen.  Arbitrary-oriented ship detection framework in optical remote-sensing images.  IEEE Geoscience and Remote Sensing Letters, 15(6):937–941, 2018\.
  * [19] Zikun Liu, Jingao Hu, Lubin Weng, and Yiping Yang.  Rotated region based CNN for ship detection.  In Proceedings of the IEEE International Conference on Image Processing, pages 900–904, 2017.
  * [20] Zikun Liu, Hongzhen Wang, Lubin Weng, and Yiping Yang.  Ship rotated bounding box space for ship extraction from high-resolution optical satellite images with complex backgrounds.  IEEE Geoscience and Remote Sensing Letters, 13(8):1074–1078, 2016\.
  * [21] Zikun Liu, Liu Yuan, Lubin Weng, and Yiping Yang.  A high resolution optical satellite image dataset for ship recognition and some new baselines.  In International Conference on Pattern Recognition Applications and Methods, pages 324–331, 2017.
  * [22] Yang Long, Yiping Gong, Zhifeng Xiao, and Qing Liu.  Accurate object localization in remote sensing images based on convolutional neural networks.  IEEE Transactions on Geoscience and Remote Sensing, 55(5):2486–2498, 2017.
  * [23] Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang, Hong Wang, Yingbin Zheng, and Xiangyang Xue.  Arbitrary-oriented scene text detection via rotation proposals.  IEEE Transactions on Multimedia, 20(11):3111–3122, 2018.
  * [24] Qi Ming, Zhiqiang Zhou, Lingjuan Miao, Hongwei Zhang, and Linhao Li.  Dynamic anchor learning for arbitrary-oriented object detection.  arXiv preprint arXiv: 2012.04150, 2020.
  * [25] Xingjia Pan, Yuqiang Ren, Kekai Sheng, Weiming Dong, Haolei Yuan, Xiaowei Guo, Chongyang Ma, and Changsheng Xu.  Dynamic refinement network for oriented and densely packed object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11204–11213, 2020.
  * [26] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 779–788, 2016.
  * [27] Joseph Redmon and Ali Farhadi.  YOLO9000: better, faster, stronger.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6517–6525, 2017.
  * [28] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  In Advances in Neural Information Processing Systems, pages 91–99, 2015.
  * [29] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  FCOS: fully convolutional one-stage object detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9626–9635, 2019.
  * [30] Jinwang Wang, Wen Yang, Heng-Chao Li, Haijian Zhang, and Gui-Song Xia.  Learning center probability map for detecting objects in aerial images.  IEEE Transactions on Geoscience and Remote Sensing, pages 1–17, 2020.
  * [31] Jialian Wu, Liangchen Song, Tiancai Wang, Qian Zhang, and Junsong Yuan.  Forest R-CNN: large-vocabulary long-tailed object detection and instance segmentation.  In Proceedings of the ACM International Conference on Multimedia, pages 1570–1578, 2020.
  * [32] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge J. Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang.  DOTA: A large-scale dataset for object detection in aerial images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3974–3983, 2018.
  * [33] Yongchao Xu, Mingtao Fu, Qimeng Wang, Yukang Wang, Kai Chen, Gui-Song Xia, and Xiang Bai.  Gliding vertex on the horizontal bounding box for multi-oriented object detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(4):1452–1459, 2021.
  * [34] Xue Yang, Qingqing Liu, Junchi Yan, and Ang Li.  R3det: Refined single-stage detector with feature refinement for rotating object.  arXiv preprint arXiv: 1908.05612, 2019.
  * [35] Xue Yang, Hao Sun, Kun Fu, Jirui Yang, Xian Sun, Menglong Yan, and Zhi Guo.  Automatic ship detection in remote sensing images from google earth of complex scenes based on multiscale rotation dense feature pyramid networks.  Remote Sensing, 10(1):132, 2018.
  * [36] Xue Yang, Jirui Yang, Junchi Yan, Yue Zhang, Tengfei Zhang, Zhi Guo, Xian Sun, and Kun Fu.  Scrdet: Towards more robust detection for small, cluttered and rotated objects.  In Proceedings of the IEEE International Conference on Computer Vision, pages 8231–8240, 2019.
  * [37] Cong Yao, Xiang Bai, Wenyu Liu, Yi Ma, and Zhuowen Tu.  Detecting texts of arbitrary orientations in natural images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1083–1090, 2012.
  * [38] Jingru Yi, Pengxiang Wu, Bo Liu, Qiaoying Huang, Hui Qu, and Dimitris Metaxas.  Oriented object detection in aerial images with box boundary-aware vectors.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, page 2150–2159, 2020.
  * [39] Gongjie Zhang, Shijian Lu, and Wei Zhang.  Cad-net: A context-aware detection network for objects in remote sensing imagery.  IEEE Transactions on Geoscience and Remote Sensing, 57(12):10015–10024, 2019.
  * [40] Hongkai Zhang, Hong Chang, Bingpeng Ma, Naiyan Wang, and Xilin Chen.  Dynamic R-CNN: towards high quality object detection via dynamic training.  In Proceedings of the European Conference on Computer Vision, pages 260–275, 2020.
  * [41] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li.  Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9756–9765, 2020.
  * [42] Zenghui Zhang, Weiwei Guo, Shengnan Zhu, and Wenxian Yu.  Toward arbitrary-oriented ship detection with rotated region proposal and discrimination networks.  IEEE Geoscience and Remote Sensing Letters, 15(11):1745–1749, 2018.
  * [43] Haigang Zhu, Xiaogang Chen, Weiqun Dai, Kun Fu, Qixiang Ye, and Jianbin Jiao.  Orientation robust object detection in aerial images using deep convolutional neural network.  In Proceedings of the IEEE International Conference on Image Processing, pages 3735–3739, 2015.
  * [44] Pengfei Zhu, Longyin Wen, Dawei Du, and et al.  Visdrone-vdt2018: The vision meets drone video detection and tracking challenge results.  In Proceedings of the European Conference on Computer Vision Workshops, pages 496–518, 2018.


# Measuring arrangement and size distributions of flowing droplets in
microchannels through deep learning

Mihir Durve<EMAIL_ADDRESS>Center for Life Nano- & Neuro-Science,
Fondazione Istituto Italiano di Tecnologia (IIT), viale Regina Elena 295,
00161 Rome, Italy  Sibilla Orsini  NEST, Istituto Nanoscienze-CNR and Scuola
Normale Superiore, Piazza San Silvestro 12, Pisa, 56127, Italy  Istituto per
le Applicazioni del Calcolo del Consiglio Nazionale delle Ricerche, via dei
Taurini 19, Roma, 00185, Italy  Adriano Tiribocchi  Istituto per le
Applicazioni del Calcolo del Consiglio Nazionale delle Ricerche, via dei
Taurini 19, Roma, 00185, Italy  Andrea Montessori  Dipartimento di Ingegneria,
Università degli Studi Roma tre, via Vito Volterra 62, Rome, 00146, Italy
Jean-Michel Tucny  Center for Life Nano- & Neuro-Science, Fondazione Istituto
Italiano di Tecnologia (IIT), viale Regina Elena 295, 00161 Rome, Italy
Dipartimento di Ingegneria, Università degli Studi Roma tre, via Vito Volterra
62, Rome, 00146, Italy  Marco Lauricella  Istituto per le Applicazioni del
Calcolo del Consiglio Nazionale delle Ricerche, via dei Taurini 19, Roma,
00185, Italy  Andrea Camposeo  NEST, Istituto Nanoscienze-CNR and Scuola
Normale Superiore, Piazza San Silvestro 12, Pisa, 56127, Italy  Dario
Pisignano  NEST, Istituto Nanoscienze-CNR and Scuola Normale Superiore, Piazza
San Silvestro 12, Pisa, 56127, Italy  Dipartimento di Fisica, Università di
Pisa, Largo B. Pontecorvo 3, Pisa, 56127, Italy  Sauro Succi  Center for Life
Nano- & Neuro-Science, Fondazione Istituto Italiano di Tecnologia (IIT), viale
Regina Elena 295, 00161 Rome, Italy  Department of Physics, Harvard
University, 17 Oxford St, Cambridge, MA 02138, United States

###### Abstract

In microfluidic systems, droplets undergo intricate deformations as they
traverse flow-focusing junctions, posing a challenging task for accurate
measurement, especially during short transit times. This study investigates
the physical behavior of droplets within dense emulsions in diverse
microchannel geometries, specifically focusing on the impact of varying
opening angles within the primary channel and injection rates of fluid
components. Employing a sophisticated droplet tracking tool based on deep-
learning techniques, we analyze multiple frames from flow-focusing experiments
to quantitatively characterize droplet deformation in terms of ratio between
maximum width and height and propensity to form liquid with hexagonal
crystalline order. Our findings reveal the existence of an optimal opening
angle where shape deformations are minimal and crystal-like arrangement is
maximal. Variations of fluid injection rates are also found to affect size and
packing fraction of the emulsion in the exit channel. This paper offers
insights into deformations, size and structure of fluid emulsions relative to
microchannel geometry and other flow-related parameters captured through
machine learning, with potential implications for the design of microchips
utilized in cellular transport and tissue engineering applications.

##  I Introduction

The high-throughput generation of fluid droplets holds significant importance
in various fields, such as chemistry, biology and material science Teh  _et
al._ (2008); Seemann  _et al._ (2012); Abate  _et al._ (2010), and has found
applications in different sectors of modern industries, ranging from food
processing Muschiolik (2007); Skurtys and Aguilera (2008); He  _et al._ (2020)
and pharmaceutics for drug delivery Xu  _et al._ (2009); Vladisavljević  _et
al._ (2013); He  _et al._ (2019) to nanoparticle synthesis Dendukuri and Doyle
(2009); Walther and Müller (2013) and soft bio-materials Visser  _et al._
(2018).

Within the realm of microfluidics, the process can be realized through several
techniques, which can be categorized into three primary types based on channel
configurations: cross-flow, co-flow, and flow-focusing Anna (2016); Teh  _et
al._ (2008); Seemann  _et al._ (2012). The co-flow and flow-focusing methods,
in particular, are specifically adept at producing highly monodisperse
droplets. These tiny elements are well suited for the assembly of droplet-
based soft materials, such as foams Raven and Marmottant (2009), and are of
interest in lab-on-chip devices Dittrich and Manz (2006). In this respect, the
precise control of droplet formation with predefined volume and size
represents a formidable challenge since it hinges upon careful control of a
number of factors including channel design and manufacturing, characteristics
of the fluids (such as concentration, viscosity and surface tension), fine-
tuning of flow rates as well as fluid-structure interaction Anna (2016);
Köster  _et al._ (2008); Diotallevi  _et al._ (2009); Montessori  _et al._
(2021a, 2019); Kernakov  _et al._ (2022); Montessori  _et al._ (2022). Despite
the complexity of the process, it has been shown in a number of papers that
the confinement of droplets brings out a variety of self-assembled ordered
structures, ranging from foams to crystal-like templates with different degree
of order, that would not exist without boundaries Marmottant and Raven (2009);
Raven and Marmottant (2009); Garstecki and Whitesides (2006). While many
studies have been focused on characterizing the dynamics and arrangement of
fluid droplets in symmetric microfluidic channels in which the striction
connects the exit channel by means of fixed opening angles Amstad  _et al._
(2016); Utada  _et al._ (2005); Vecchiolla  _et al._ (2018), much less is
known about the effect of such angles on droplet morphology and subsequent
structures downstream.

This work investigates the influence of microchannel geometry on the size and
spatial arrangement of droplets within a flow-focusing apparatus. Advances in
manufacturing methodologies Su  _et al._ (2023) for microfluidic devices,
coupled with innovative computer vision algorithms, have facilitated the
analysis of exceptionally high-quality data, allowing for the precise
quantification of droplet characteristics across diverse configurations.
Leveraging DropTrack Durve  _et al._ (2021a, b, 2022, 2023), a previously
established deep leaning-based droplet tracking tool, we quantify parameters
related to droplet deformations, size distribution, packing density and
arrangement under the confined environment of a microfluidic channel. We find
that the geometry of the microchannel combined with the complex structure of
the fluid flows critically affect the droplet-droplet interactions, yielding
an intricate picture where the opening angle of the channel and injection
rates of the fluids control size and arrangement in a highly non-trivial
manner. Indeed, keeping fixed the rates of oil (dispersed phase) and water
(continuous phase), both deformation and average size of the droplets lessen
for increasing values of opening angle. The latter is also found to affect
droplet order, which exhibits an optimal hexagonal arrangement for a
restricted range of opening angles. However, increasing the oil injection rate
generally disrupts such order producing weakly-packed small droplets,
essentially because of the high shear flows in the channel. Modification of
the water rate has a milder impact on drop size, whereas it can considerably
alter the resulting structure of the emulsion downstream. The complex scenario
emerging from these findings suggests that predicting the structural and
mechanical properties of a fluid emulsion in a highly confined regime remains
a challenge if solely based on the control of a limited set of physical
parameters. Machine learning (ML) methods (potentially in combination with
computer simulations) can effectively bridge this gap, especially if tested on
different lab-on-chip platforms to minimize the dependence on a specific
training dataset.

The paper is structured as follows. In the next section, we illustrate the
details of the experiments and the DropTrack algorithm. Afterwards, we present
the results which contain a discussion about droplet deformability, size
distribution, degree of order and packing fraction in channels with varying
opening angles and oil/water injection rates. Some final considerations and
potential perspectives close the manuscript.

##  II Experimental Details and tracking algorithm structure

###  II.1 Materials

The materials used in the experiment are the following. The E-Shell®600
(EnvisionTEC) photocurable material was used as the pre-polymer. Also,
Brilliant Black BN pigment was purchased from ABCR GmbH, Tween®20 and
isopropyl alcohol were obtained from Sigma-Aldrich, while sunflower seed oil
from Santa Cruz Biotechnology.

###  II.2 Device for droplets generation

A flow-focusing microfluidic junction was designed to produce water-in-oil
(W/O) emulsion droplets (Figure 1a). The devices have one central inlet for
the dispersed phase (water solution) and two side inlets for the continuous
phase (sunflower oil). The latter channels form a 45° angle with respect to
the central one. The inlet channels have a width of 500 $\mu$m. The flow-
focusing junction is connected to an expansion channel with an opening angle
($\alpha$) of 30, 45, 60 or 90 degrees, respectively. The expansion channel
has a width of 2 mm and a length of 9 mm. All channels have a depth of 500
$\mu$m. To prevent the presence in the channel of residual pre-polymer
polymerized during the printing process, the channels were left open in the
central area of the device (i.e. they were designed with one edge positioned
on the surface of the device). In order to seal them, a 400 $\mu$m thick layer
was designed as a cap for the channels. Either the inlet or outlet channels
were connected to four cylindrical apertures with internal 2.2 mm diameter and
3.8 mm height, which host conical shaped hollow connectors used for inserting
plastic tubes.

Figure 1: a) Schematic representation of the flow-focusing junction and the
expansion channel designed for the generation of W/O emulsion. b) Sketch of
the DLP 3D printing system. c)-f) Illustration of the fabrication steps of the
microfluidic device. First, the device with open channels and the sealing
layer are printed (c) and bonded through UV-light irradiation (d). Afterwards,
two glass coverslips are bonded on the top and bottom surfaces of the device
by UV light irradiation and photopolymerization of an interposed layer of pre-
polymer (e). Finally, plastic tubes are connected to the inlets and outlet
apertures by means of 3D printed connectors (f). g-h) Photographs of the
device after step d), (g), and of the final device (h). Scale bars: 9 mm. i-l)
Bright field optical microscope images of the inlet of the expansion chamber
with opening angle: 30 (i), 45 (j), 60 (k) and 90 (l) degrees. Scale bar: 1
mm.

###  II.3 Device fabrication

The device, the hollow connectors and the sealing layer were printed by a
digital light processing (DLP) system (MicroPlus, HD EnvisionTEC) equipped
with an UV light source with emission wavelength of 405 nm and intensity of 2
mW cm-2 (Figure 1b). The objects to be printed were sliced in 50 $\mu$m thick
layers. A UV light exposure time of 4.8 s was used for each layer. After the
printing process, the device with open channels and the sealing layer were
bonded through exposure to the emission of a UV light lamp (wavelength: 365
nm, intensity: 0.7 mW cm-2, exposure time: 20 s), as illustrated in Figure
1c-d. Afterwards, two glass coverslips (Bio-Optica, 22$\times$22 mm2) were
bonded to the top and bottom surface of the device, through the
photopolymerization of a layer of E-Shell®600 interposed between the glass
coverslips and the device surfaces (Figure 1e). To this aim, the UV lamp (365
nm) was used (exposure time: 20 s). The resulting devices were washed with
isopropyl alcohol and dried in a nitrogen flow. Finally, the hollow connectors
and flexible plastic tubes were connected to the device (Figure 1f). A example
of the realized device is shown in Figure 1g-h, whereas optical microscope
images of the inlet of the expansion channel with various opening angles are
shown in Figure 1i-l, respectively.

###  II.4 Droplet formation

For obtaining W/O emulsions, Tween®20 (1 mg/ml) and Brilliant Black (7 mg/mL)
were dissolved in deionized water as the surfactant and the black pigment,
respectively. In particular, the latter was exploited to better visualize the
emulsion formation and the droplets packing in the expansion channel. The
flows of the continuous and dispersed phases through the device were
controlled by a high-precision twin syringe pump (Harvard Apparatus). The
droplets were imaged by using a stereo microscope (MZ 16 FA, Leica,
illumination in transmission mode) and a camera (Fastcam APX RS, Photron, 3000
frames per second). The sequences of images were stored as AVI video files.
Three different sequences of images were acquired for each set of experimental
data.

###  II.5 DropTrack algorithm

Here we shortly recap the main features of DropTrack, while we refer to the
extensive literature for further details Durve  _et al._ (2021a, b, 2022,
2023). The DropTrack software is based upon the amalgamation of YOLO (You Only
Look Once) Redmon  _et al._ (2016) and DeepSORT (Deep Simple Online and
Realtime Tracking) Durve  _et al._ (2021a) algorithms, adeptly tailored to the
specialized task of identifying and tracking droplets within image sequences.
Through customization and training, DropTrack has been calibrated to precisely
recognize droplets within images and extract their trajectories across
consecutive frames. Consequently, the application furnishes comprehensive
data, encompassing the dynamic paths of individual droplets plus the
associated width and height dimensions of the bounding boxes encapsulating
them. A single representative instance of DropTrack’s output is provided in
the supplemental material (see movieA.mp4). DropTrack was employed to analyze
a total of forty-four such videos depicting the flow of droplets within
diverse microchannel geometries. DropTrack’s output, in conjunction with
temporal data, facilitate the extraction of many physical observables
pertaining to the size and spatial disposition of the droplets, as we report
in the next sections.

In this investigation, we systematically explore the variability in droplet
volume within the context of four distinct geometric microchannels,
characterized by varying opening angles $\alpha$ ranging from 30 to 90 degrees
(refer to Fig. 1(a)). Each experiment undergoes scrutiny over a duration of
0.68 seconds, yielding a collection of 2044 frames in video format. These ones
are then analysed by DropTrack, which facilitates the generation of bounding
boxes encircling individual droplets, with each droplet being assigned a
unique identifier. The size of each droplet is finally approximated by an
ellipse encompassed by the aforementioned bounding box.

##  III Results

###  III.1 Measuring the deformability as a function of the opening angle

Figure 2: Droplet aspect ratio for various microchannel geometries. The aspect
ratio is measured as the ratio of width to height of the droplet. The inset
labels report water (w) and oil (o) injection rates in $\mu l$ per min, while
the images show droplets with maximal (top) and minimal (bottom) deformation
for two different values of $\alpha$. Each droplet is labelled by a colored
bounding box and a number. Increasing the aperture angle diminishes the
deformation, basically because of a reduction of the packing density
downstream.

As the liquid droplets traverse the main channel, shape deformations, mainly
attributed to the presence of preceding droplets and the constrained spatial
environment, start to occur, as illustrated in Fig.1a. Maximum deformation of
these droplets is discerned primarily at the initial segment of the exit
channel since, as they progress downstream, spatial confinement reduces and an
equilibrium configuration is gradually attained. The extent of droplet
deformability can be quantified in terms of the ratio of width to height
within a bounding box, as identified through the DropTrack software.

In Fig.2 we show the average maximum aspect ratio measured within each frame
for several values of water and oil injection rates in microchannels of
different opening angles $\alpha$ (as detailed in Section II). On a general
basis, as $\alpha$ increases, there is a propensity for a decrease in
deformability. This is because augmenting $\alpha$ expands the available space
at the inception of the primary channel, thus reducing local packing density
and, consequently, shape deformations. However, the interrelationship between
these parameters follows a complex pattern. Indeed, the trend deviates from
the observations made for opening angles exceeding $\alpha_{c}\simeq
60^{\circ}$ (essentially regardless of the water/oil injection rates explored
in the present study) because, in these cases, the opening becomes
sufficiently wide to allow the droplets to relax towards a quasi-spherical
shape. Notably, the existence of an optimal value of $\alpha$ emerges as a key
factor for the control of droplet morphology during transport, as discussed in
the next section.

###  III.2 Measuring the size distribution

Since the experimental setup employs a camera capturing two-dimensional
images, in Fig.3a we depict the distribution of droplet area resulting from
four distinct microchannel geometries, keeping constant injection rates of oil
and water across all configurations. The distribution displays a bimodal
nature, delineating two discernible groups: this signifies a marked transition
in droplet sizes when $\alpha$ exceeds 60 degrees. Note that, within the
subgroup constituted by opening angles $30,45$ and $60$ degrees, the extent of
the distribution spread presents a nuanced pattern with a mean around
$4-5\times 10^{-8}$ $\mu m^{2}$, while for larger angles it shrinks towards
lower sizes, basically because the emulsion turns almost monodisperse (with
smaller circular droplets) under weaker confinement.

Figure 3: (a) Droplet size distribution for four different values of the
opening angle $\alpha$ (as shown in Fig.1a). The water injection rate is 300
$\mu l$ per min while the oil injection rate is 150 $\mu l$ per min. If
$\alpha$ exceeds $60$ degrees, the droplet size undergoes a sharp decrease.
(b)-(c) Droplet size distribution for three oil injection rates (b) and four
water injection rates (c). The opening angle is $\alpha=60^{\circ}$.
Increasing the oil rate diminishes droplet size; this is because the high
shear at the nozzle facilitates the rupture of the jet leading to smaller
droplets. On the contrary, changing the water injection rate mildly affects
the size. Water (w) and oil (o) injection rates are in $\mu l$ per min.

Interestingly, the size distribution of the droplets is also significantly
influenced by the rates at which oil and water are injected into the system.
In Fig.3b-c, we show the measured droplet size distribution for various oil
(b) and water (c) injection rates (while maintaining a constant rate of the
other component) and for $\alpha=60^{\circ}$. Notably, the droplets exhibit a
considerable reduction in size as the oil injection rate augments while an
elevation of the variance as the rate decreases. This is essentially due to
the increasing shear at the orifice, an effect that favours the rupture of the
water jet and the formation of smaller droplets. On the contrary, varying the
water injection rate causes minor effects on droplet size (Fig.3c). In this
case, the mean values remain relatively consistent (around $5-6\times 10^{-8}$
$\mu m^{2}$) across different rates, while the distributions display an
amplified degree of spread.

In summary, the results discussed so far show that, while the droplet
deformation can be controlled, with good accuracy, solely by a geometrical
feature of the device (i.e. the aperture angle), the size exhibits a more
intricate behavior, in which both device design and injection rates of fluids
play a fundamental role.

###  III.3 Droplet ordering within the main channel

Alongside discernible variations of droplet dimensions and configurations in
different microchannel geometries, our results also show that, once in the
channel, the droplets attain a stable crystal-like arrangement whose degree of
organization depends, once again, on the opening angle as well as on the
injection rates of the fluids. To achieve this, we compute the hexatic order
parameter, denoted as G3, and subsequently construct its probability
distribution function Montessori  _et al._ (2021a). Such distribution is
graphically represented with respect to the measured G3 values, defined as
$G3=|\cos(3\theta_{jik})|$. Here, $\theta_{jik}$ represents the angle formed
by all droplet triads while maintaining the central droplet $i$ as a fixed
reference point and neighboring droplets $j$ and $k$ positioned immediately
adjacent to droplet $i$, as depicted in the inset of Fig.4 (a). This
formulation of G3, which converges towards unity when the angle $\theta_{jik}$
assumes multiples of $\pi/3$, serves as a robust metric for discerning whether
the neighboring droplets of a given droplet $i$ conform to a hexagonal lattice
arrangement.

Figure 4: Probability Distribution Function (PDF) of G3 order parameter for
various opening angles $\alpha$ (a) 30, (b) 45, (c) 60 and (d) 90 degrees. The
water injection rate is 300 $\mu l$ per min and the oil one is 150 $\mu l$ per
min. An optimal hexagonal arrangement is found for $\alpha\simeq 60^{\circ}$
where the distribution displays a pronounced peak at $1$. The fat tails
observed for other values of $\alpha$ indicate a limited crystal-like order.
Inset in (a): schematic illustration of the geometry of the droplets array
used for the calculation of G3.

The probability distribution function of G3 values, measured across four
distinct geometries, is presented in Fig.4. Interestingly, the microchannel
characterized by an opening angle of $\alpha=60^{\circ}$ exhibits a prominent
peak with G3 values reaching unity, thus suggesting that a significant
proportion of the angles (as measured between droplet triads $jik$) closely
approximate multiples of $\pi/3$. Also, the droplets manifest an intermediate
size distribution for this specific geometry (as depicted in Figure 3), an
indication that this one represents an optimal condition for self-organization
into a hexagonal lattice. In contrast, smaller droplets observed at an opening
angle of $\alpha=90^{\circ}$ possess greater freedom in their movement,
thereby disrupting the hexagonal order, while larger ones with an opening
angle of $\alpha=45^{\circ}$ encounter spatial constraints that limit their
ability to achieve hexagonal symmetry.

Figure 5: Distribution of G3 order parameter for various water (w) and oil (o)
injection rates (in $\mu l$ per min). The opening angle $\alpha$ is kept
constant at 60 degrees. The insets show equilibrium droplet configuration down
the main channel. An optimal crystal-like arrangement (green plot, top row and
red plot, bottom row) is found for droplets of intermediate sizes displaying
reduced shape fluctuations. Illustrative images of droplets in the main
channel corresponding to the water and oil injection rates are shown within
the insets.

As previously mentioned, the propensity for hexagonal lattice formations is
also affected by the water/oil injection rates. In Fig.5 we showcase the G3
distribution obtained when either oil (top row) or water (bottom row) rates
are varied, for a fixed microchannel geometry. We find that the configurations
characterized by intermediate droplet sizes (as depicted in the green plot of
Fig.3b and the red one of Fig.3c) showcase the most conspicuous G3 order
parameter peak (see the central plot, top row, and left plot, bottom row, of
Fig.5), likely because under these conditions the emulsion attains a suitable
balance between droplet size and degree of confinement. Indeed, this
phenomenon is particularly pronounced when the system exhibits minimal
fluctuations in droplet size distributions, as illustrated for example in the
red plot of Fig.3c and in the corresponding G3 distribution in the left lower
panel of Fig.5.

These observations collectively suggest that the emergence of a crystalline
order depends, in a highly non-trivial way, on an intricate interplay between
droplet size, microchannel geometry, and fluid injection rates.

###  III.4 Hexagonal structures and packing fraction

In the previous section, the calculation of the G3 order parameter takes into
account all droplets in the channel, thus including the ones near the walls
where perfect hexagonal symmetry is inevitably absent. Here, we assess the
proportion of droplets having precisely six neighboring droplets and
exhibiting a hexatic order parameter exceeding 0.98 for all triads, thus
inherently excluding droplets located at the boundaries.

The variation in the fraction of droplets satisfying these criteria is
visualized in Fig.6a across diverse experimental parameters. Unlike the
previous cases, here clear and definitive trends are less discernible when
considering either experiments with different opening angles or microchannel
geometries featuring varying water and oil injection rates. In broad terms,
the fraction of hexagonal structures in the bulk of the microchannel ranges
approximately from $0.1$ to $0.35$, an indication that, in the best scenario,
only $\sim 35\%$ of droplets self-organize in a perfect crystal-like
configuration. Nonetheless, a value equal to $0.35$ can be considered good
enough considering the stringent criteria employed to identify these hexagonal
structures (which necessitate both exactly six neighbors and a G3 higher than
$0.98$) and the limited control on their formation achieved by solely changing
either the injection rates of fluids or the geometrical setup. Finally, the
higher fractions of hexagonal structures observed in larger systems (see the
results at $\alpha=90^{\circ}$) is somewhat expected given the inherent nature
of hexagonal packing.

Figure 6: (a) The fraction of hexagonal structures within the main channel for
various combinations of experimental parameters. (b) The droplet packing
fraction within the main channel for various combinations of experimental
parameters. Its values are mainly controlled by the oil/water injection rates,
while they are very weakly affected by the opening angle. The inset labels
show the water injection rate (w) and the oil injection rate (o) in $\mu l$
per min. Illustrative images of droplets in the main channel corresponding to
the maximum and minimum values of the observables are shown within the insets.

A deeper insight can be gained by computing the droplet packing fraction,
defined as the ratio of the cumulative area occupied by droplets (which are
approximated as ellipses enclosed within identified bounding boxes) within the
channel with respect to the total channel area. In Fig.6b we show the packing
fraction in four distinct geometries and under varying flow conditions. It
generally ranges from $\sim 0.3$ to $\sim 0.8$, a variation mainly due to flow
rate changes rather than opening angles. More specifically, lower values are
found for higher oil injection rates (see, for example, the blue circles in
Fig.6b), basically because the large shear of the dispersed phase hinders the
formation of highly packed configurations of droplets. As the oil rate turns
lower, packing fractions stabilize around 0.8 for all values of $\alpha$
explored in the experiment. It is finally worthwhile to note that these
packing fraction measurements are quickly estimated by DropTrack in real-time,
although their precision can slightly diminish for droplets characterized by
heightened deformability.

##  IV Conclusion

In this work, we study the effects of opening angle and injection rates of oil
(continuous phase) and water (dispersed phase) on droplet sizes and resulting
emulsion structure within a fabricated microchip. The physics is captured by a
number of statistical properties, such as aspect ratio, size distribution and
hexatic order parameter computed using DropTrack, a ML based image analysis
tool successfully adopted to identify and track droplets in microfluidic
experiments. This technique, besides providing accurate and reliable results
(as also demonstrated by previous works Durve  _et al._ (2021a, b, 2022,
2023)), considerably expedite the analysis, which would typically be a
challenging task owing to the prohibitive labor-intensive nature of
traditional methodologies.

Our results show that, while the droplet aspect ratio is found to diminish for
increasing opening angles and essentially regardless of injection rates of
fluids, the size distribution of the droplets as well as their arrangement in
the channel exhibit a more intricate scenario. For fixed values of injection
rates of dispersed and continuous phase, we find a suitable aperture angle
$\alpha_{c}$ (approximately equal to $60$ degrees in our setup) where large
part of the emulsion self-organizes into a crystalline structure, very likely
because of an optimal balance between droplet size and degree of confinement.
For $\alpha>\alpha_{c}$, droplet size turns considerably smaller and uniform,
basically because shape deformations become milder under weak confinement,
thus the degree of order is found to decrease. Changing the injection rates of
the fluids (keeping constant the aperture angle) further enriches the picture.
Indeed, increasing the oil injection rate (continuous phase) diminishes the
droplet size due to the high shear at the orifice; on the contrary, augmenting
the water injection rate (dispersed phase) negligibly affects the medium value
while amplifying the spread of the size distribution. Our findings also
suggest that the injection rates of both fluids can be properly tuned to
control the formation of ordered structures in the channel. In particular, low
rates of the dispersed phase favour regular hexagonal configurations with a
packing fraction varying from $\sim 0.6$ to $\sim 0.8$ regardless of the
confinement conditions, while higher values compromise the stability of such
super-structures.

The measurement of droplet size, structural characteristics, and deformability
holds paramount significance in different contexts ranging from material
science for the manufacturing of scaffolds of tissues Guevorkian  _et al._
(2010); Douezan  _et al._ (Proc. Natl. Acad. Sci. U.S.A.); Costantini  _et
al._ (2014), to the biological domain for understanding the behavior of cell
clusters crossing physiological constrictions Bogdan  _et al._ (2022);
Tiribocchi  _et al._ (2023); Montessori  _et al._ (2021b); Au  _et al._
(2016), up to the pharmaceutics for improving design and functioning of drug
delivery products Pays  _et al._ (2002); Pontrelli  _et al._ (2020). However,
achieving precise control of droplet features (such as size distribution while
adhering to defined deformability) under highly confined regimes is still a
difficult task in the realm of microchip design. In this investigation, we
have elucidated the intricate interdependency among dimensions, spatial
arrangement, and deformability characteristics of droplets as they flow within
the primary channel of a flow-focusing microchip. This has been done using the
deep-learning software DropTrack which is proven, once more, a powerful
platform to understand the physics of these complex systems and to potentially
predict their behavior. Its demonstrated applicability to various lab’s data
pertaining to different microfluidic experiments considerably limits the risk
of overfitting, which occurs when models perform well within the developer’s
setup but poorly in others. This is particularly important in fluid dynamics,
where the complexity of the parameter space involving different scales as well
as the details of the device manufacturing may crucially affect performance
and results. In addition, its capability to recognize patterns and non-trivial
dependencies often inaccessible to experiments is a crucial step to strengthen
the model and enhance the automation of the identification/tracking procedure.

Further in-depth inquiries are imperative to address a number of questions of
relevance for experiments and theory, comprising the use of ML tools in
microfluidics. On the practical side, the ability to control flow rates,
volume fractions and interfacial properties to augment the production of
droplet-based soft materials and to standardize their manufacturing procedure
remains a challenge. Much of the complexity raises from droplet deformability,
which introduces non-linearities in the otherwise linear Stokes flows,
together with variations of the channel geometry, which often affect the
dynamics in unexpected ways Anna (2016). Alongside these macroscopic effects,
a careful assessment of the near-contact interactions governing the physics at
the scales of the fluid interfaces as well as thermal properties are crucial
to control the mechanical stability of the materialD’Orazio and Succi (2003).
This highly non-trivial parameter space results in a multifarious range of
flowing patterns whose features are exceptionally difficult to predict.

The integration of ML methods to computer simulations would surely contribute
to tackle the complex fluid dynamics of these multiphase flows as well as to
optimize design and testing, potentially leading a higher rate of experimental
success and easier commercialization of microfluidic chips McIntyre  _et al._
(2022); Lashkaripour  _et al._ (2021). However, further work is needed to
build advanced ML techniques capable of quantifying, for example, the
evolution of droplet shapes and break-up conditions observed under different
flow rates, especially in systems composed of tightly packed droplets where
morphological deformations significantly depart from elliptical-like
geometries. Indeed, a question of high relevance concerns the requisite
microchip dimensions yielding droplets of desired size and structural
attributes. These improvements could considerably extend the applicability of
ML tools to other soft materials such as foam, cells and tissues. Finally,
much efforts should be addressed to scale up the operation of ML methods to
lab-on-chip platforms, in order to minimize modelling changes caused by the
use of different protocols or training datasets.

###### Acknowledgements.

M.D. and S.O. contributed equally to this work. The authors acknowledge
funding from the European Research Council Grant Agreement No. 739964 (COPMAT)
and ERC-PoC2 grant No. 101081171 (DropTrack). M.L. acknowledges the support of
the Italian National Group for Mathematical Physics (GNFM-INdAM). A.C. and
D.P. acknowledge the support of the European Union by the Next Generation EU
project ECS00000017 ‘Ecosistema dell’Innovazione’ Tuscany Health Ecosystem
(THE, PNRR, Spoke 4: Nanotechnologies for diagnosis and therapy). We
gratefully acknowledge the HPC infrastructure and the Support Team at
Fondazione Istituto Italiano di Tecnologia.

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

##  V Conflict of Interest Statement

The authors have no conflicts to disclose.

##  VI CRediT (Contribution Role Taxonomy) Statement

Conceptualization – M.D., M.L., S.O., A.C., J-M. T., A.M., S.S. (equal)
Data Curation – M.D., S.O., A.C. (equal)
Formal Analysis – M.D., A.T., M.L. (equal)
Funding Acquisition – A.C. D.P., S.S.
Methodology – M.D., S.O., A.C., A.T., M.L. (equal)
Project Administration – S.S.
Software – M.D., M.L., A.T., A.M. (equal)
Supervision – S.S., A.C., D.P.
Visualization – M.D., A.T., M.L.
Writing/Original Draft Preparation – A.T., M.D., J-M. T., A.C., S.O.
Writing/Review & Editing – All authors (equal)

## References

  * Teh  _et al._ (2008) S. Y. Teh, R. Lin, L. H. Hung, and A. P Lee, “Droplet microfluidics,” Lab on a Chip 8, 198–220 (2008).
  * Seemann  _et al._ (2012) R. Seemann, M. Brinkmann, T. Pfohl, and S. Herminghaus, “Droplet based microfluidics,” Rep. Prog. Phys. 75, 016601 (2012).
  * Abate  _et al._ (2010) A. R. Abate, T. Hung, P. Mary, J. J. Agresti, and D. A. Weitz, “High-throughput injection with microfluidicsusing picoinjectors,” Proc. Natl. Acad. Sci. U.S.A. 107, 19163–19166 (2010).
  * Muschiolik (2007) G. Muschiolik, “Multiple emulsions for food use,” Curr. Opin. Colloid. Interface Sci. 12, 213–220 (2007).
  * Skurtys and Aguilera (2008) O. Skurtys and J. M. Aguilera, “Applications of microfluidic devices in food engineering,” Food Biophys. 3, 1–15 (2008).
  * He  _et al._ (2020) Shan He, Nikita Joseph, Shilun Feng, Matt Jellicoe, and Colin L. Raston, “Application of microfluidic technology in food processing,” Food Funct. 11, 5726–5737 (2020).
  * Xu  _et al._ (2009) Q. B. Xu, M. Hashimoto, T. T. Dang, T. Hoare, D. S. Kohane, G. M. Whitesides, R. Langer, and D. G. Anderson, “Preparation of monodisperse biodegradable polymer microparticles using a microfluidic flow-focusing device for controlled drug delivery,” Small 13, 1575–1581 (2009).
  * Vladisavljević  _et al._ (2013) G. T. Vladisavljević, N. Khalid, M. A. Neves, T. Kuroiwa, M. Nakajima, K. Uemura, S. Ichikawa, and I. Kobayashi, “Industrial lab-on-a-chip: Design, applications and scale-up for drug discovery and delivery,” Adv. Drug Del. Rev. 65, 1626–1663 (2013).
  * He  _et al._ (2019) Fan He, Mao-Jie Zhang, Wei Wang, Quan-Wei Cai, Yao-Yao Su, Zhuang Liu, Yousef Faraj, Xiao-Jie Ju, Rui Xie, and Liang-Yin Chu, “Designable polymeric microparticles from droplet microfluidics for controlled drug release,” Advanced Materials Technologies 4, 1800687 (2019), https://onlinelibrary.wiley.com/doi/pdf/10.1002/admt.201800687 .
  * Dendukuri and Doyle (2009) D. Dendukuri and P. S. Doyle, “The synthesis and assembly of polymeric microparticles using microfluidics,” Adv. Mater. 21, 4071–4086 (2009).
  * Walther and Müller (2013) A. Walther and A. H. E. and Müller, “Janus particles: Synthesis, self-assembly, physical properties, and applications,” Chem. Rev. 113, 5194–5261 (2013).
  * Visser  _et al._ (2018) C. W. Visser, T. Kamperman, L P. Karbaat, D. Lohse, and M Karperien, “In-air microfluidics enables rapid fabrication of emulsions, suspensions, and 3d modular (bio)materials,” Science Advances 4, 1 (2018).
  * Anna (2016) S. L. Anna, “Droplets and bubbles in microfluidic devices,” Annu. Rev. Fluid Mech. 48, 285–309 (2016).
  * Raven and Marmottant (2009) J. P. Raven and P. Marmottant, “Microfluidic crystals: Dynamic interplay between rearrangement waves and flow,” Phys. Rev. Lett. 102, 084501 (2009).
  * Dittrich and Manz (2006) P. S. Dittrich and A. Manz, “Lab-on-a-chip: microfluidics in drug discovery,” Nature Reviews Drug Discovery 5, 210–218 (2006).
  * Köster  _et al._ (2008) S. Köster, F.E. Angilé, H. Duan, J. J. Agresti, A. Wintner, C. Schmitz, A. C. Rowat, C. A. Merten, D. Pisignano, A. D. Griffiths, and D. A. Weitz, “Drop-based microfluidic devices for encapsulation of single cells,” Lab Chip 8, 1110–1115 (2008).
  * Diotallevi  _et al._ (2009) F. Diotallevi, L. Biferale, S. Chibbaro, A. Lamura, G. Pontrelli, M. Sbragaglia, S. Succi, and F. Toschi, “Capillary filling using lattice boltzmann equations: The case of multi-phase flows,” Eur. Phys. J. Special Topics 166, 111–116 (2009).
  * Montessori  _et al._ (2021a) A. Montessori, A. Tiribocchi, M. Lauricella, F. Bonaccorso, and S. Succi, “Mesoscale modelling of droplets’ self-assembly in microfluidic channels,” Soft Matter 17, 2374–2483 (2021a).
  * Montessori  _et al._ (2019) A. Montessori, M. Lauricella, A. Tiribocchi, and S. Succi, “Modeling pattern formation in soft flowing crystals,” Phys. Rev. Fluids 4, 072201(R) (2019).
  * Kernakov  _et al._ (2022) P. Kernakov, S. Litvinov, and P. Koumoutsakos, “Computing foaming flows across scales: From breaking waves to microfluidics,” Science Advances 8, 5 (2022).
  * Montessori  _et al._ (2022) A. Montessori, M. Lauricella, and A. Tiribocchi, “Computational droplets: Where we stand and how far we can go,” EPL 138, 67001 (2022).
  * Marmottant and Raven (2009) P. Marmottant and J. P. Raven, “Microfluidics with foams,” Soft Matter 5, 3385–3388 (2009).
  * Garstecki and Whitesides (2006) P. Garstecki and G. M. Whitesides, “Flowing crystals: Nonequilibrium structure of foam,” Phys. Rev. Lett. 97, 024503 (2006).
  * Amstad  _et al._ (2016) E. Amstad, M. Chemama, M. Eggersdorfer, L. R. Arriaga, M. P. Brenner, and D. A. Weitz, “Robust scalable high throughput production of monodisperse drops,” Lab on a Chip 16, 4163–4172 (2016).
  * Utada  _et al._ (2005) A. S. Utada, E. Lorenceau, D. R. Link, P. D. Kaplan, H. A. Stone, and D. A. Weitz, “Monodisperse double emulsions generated from a microcapillary device,” Science 308, 537–541 (2005).
  * Vecchiolla  _et al._ (2018) D. Vecchiolla, V. Giri, and S. L. Biswal, “Bubble-bubble pinch-off in symmetric and asymmetric microfluidic expansion channels for ordered foam generation,” Soft Matter 14, 9312–9325 (2018).
  * Su  _et al._ (2023) Ruitao Su, Fujun Wang, and Michael C. McAlpine, “3d printed microfluidics: advances in strategies, integration, and applications,” Lab Chip 23, 1279–1299 (2023).
  * Durve  _et al._ (2021a) M. Durve, F. Bonaccorso, A. Montessori, M. Lauricella, A. Tiribocchi, and S. Succi, “A fast and efficient deep learning procedure for tracking droplet motion in dense microfluidic emulsions,” Phil. Trans. Roy. Soc. A 379, 2208 (2021a).
  * Durve  _et al._ (2021b) M. Durve, F. Bonaccorso, A. Montessori, M. Lauricella, A. Tiribocchi, and S. Succi, “Tracking droplets in soft granular flows with deep learning techniques,” Eur. Phys. J. Plus 136, 864 (2021b).
  * Durve  _et al._ (2022) M. Durve, A. Tiribocchi, F. Bonaccorso, A. Montessori, M. Lauricella, M. Bogdan, J. Guzowski, and S. Succi, “Droptrack—automatic droplet tracking with yolov5 and deepsort for microfluidic applications,” Phys. Fluids 34, 082003 (2022).
  * Durve  _et al._ (2023) M. Durve, S. Orsini, A. Tiribocchi, A. Montessori, J. M. Tucny, M. Lauricella, A. Camposeo, D. Pisignano, and S. Succi, “Benchmarking yolov5 and yolov7 models with deepsort for droplet tracking applications,” Eur. Phys. J. E 46, 32 (2023).
  * Redmon  _et al._ (2016) J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 779–788 (2016).
  * Guevorkian  _et al._ (2010) K. Guevorkian, M. J. Colbert, M. Durth, S. Dufour, and F. and Brochard-Wyart, “Aspiration of biological viscoelastic drops,” Phys. Rev. Lett. 104, 218101 (2010).
  * Douezan  _et al._ (Proc. Natl. Acad. Sci. U.S.A.) S. Douezan, K. Guevorkian, R. Naouar, S. Dufour, D. Cuvelier, and F. Brochard-Wyart, “Spreading dynamics and wetting transition of cellular aggregates,” 2011 108, 7315 (Proc. Natl. Acad. Sci. U.S.A.).
  * Costantini  _et al._ (2014) M. Costantini, C. Colosi, J. Guzowski, A. Barbetta, J. Jaroszewicz, W. Świeszkowski, M. Dentini, and P. Garstecki, “Highly ordered and tunable polyhipes by using microfluidics,” J. Mater. Chem. B 2, 2290 (2014).
  * Bogdan  _et al._ (2022) M. Bogdan, A. Montessori, A. Tiribocchi, F. Bonaccorso, M. Lauricella, L. Jurkiewicz, S. Succi, and J. Guzowski, “Stochastic jetting and dripping in confined soft granular flows,” Phys. Rev. Lett. 128, 128001 (2022).
  * Tiribocchi  _et al._ (2023) A. Tiribocchi, M. Durve, M. Lauricella, A. Montessori, D. Marenduzzo, and S. Succi, “The crucial role of adhesion in the transmigration of active droplets through interstitial orifices,” Nature Commun. 14, 1096 (2023).
  * Montessori  _et al._ (2021b) A. Montessori, A. Tiribocchi, M. Bogdan, F. Bonaccorso, M. Lauricella, J. Guzowski, and S. Succi, “Translocation dynamics of high-internal phase double emulsions innarrow channels,” Langmuir 37, 9026–9033 (2021b).
  * Au  _et al._ (2016) S. H. Au, B. D. Storey, J. C. Moore, Q. Tang, Y. L. Chen, S. Javaid, A. F. Sarioglu, R. Sullivan, M. W. Madden, R. O’Keefe, D. A. Haber, S. Maheswaran, D. M. Langenau, S. L. Stott, and M. Toner, “Clusters of circulating tumor cells traverse capillary-sized vessels,” Proc. Natl. Acad. Sci. U.S.A. 113, 4947 (2016).
  * Pays  _et al._ (2002) K. Pays, J. Giermanska-Kahn, B. Pouligny, J. Bibette, and F. Leal-Calderon, “Double emulsions: how does release occur?” J. Control. Release 79, 193 (2002).
  * Pontrelli  _et al._ (2020) G. Pontrelli, E. J. Carr, A. Tiribocchi, and S. Succi, “Modeling drug delivery from multiple emulsions,” Phys. Rev. E 102, 023114 (2020).
  * D’Orazio and Succi (2003) Annunziata D’Orazio and Sauro Succi, “Boundary conditions for thermal lattice boltzmann simulations,” in  _Computational Science — ICCS 2003_ , edited by Peter M. A. Sloot, David Abramson, Alexander V. Bogdanov, Jack J. Dongarra, Albert Y. Zomaya, and Yuriy E. Gorbachev (Springer Berlin Heidelberg, Berlin, Heidelberg, 2003) pp. 977–986.
  * McIntyre  _et al._ (2022) D. McIntyre, A. Lashkaripour, P. Fordyce, and D. Densmore, “Machine learning for microfluidic design and control,” Lab on a Chip 22, 2925 (2022).
  * Lashkaripour  _et al._ (2021) A. Lashkaripour, C. Rodriguez, N. Mehdipour, R. Mardian, D. McIntyre, L. Ortiz, J. Campbell, and D. Densmore, “Machine learning enables design automation of microfluidic flow-focusing droplet generation,” Nature Commun. 12, 25 (2021).