# Domain Adaptive Object Detection for Autonomous Driving under Foggy Weather

Jinlong Li1, Runsheng Xu2, Jin Ma1, Qin Zou3, Jiaqi Ma2, Hongkai Yu1
1 Cleveland State University, 2 University of California, Los Angeles, 3 Wuhan
University
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author

###### Abstract

Most object detection methods for autonomous driving usually assume a
consistent feature distribution between training and testing data, which is
not always the case when weathers differ significantly. The object detection
model trained under clear weather might be not effective enough on the foggy
weather because of the domain gap. This paper proposes a novel domain adaptive
object detection framework for autonomous driving under foggy weather. Our
method leverages both image-level and object-level adaptation to diminish the
domain discrepancy in image style and object appearance. To further enhance
the model’s capabilities under challenging samples, we also come up with a new
adversarial gradient reversal layer to perform adversarial mining for the hard
examples together with domain adaptation. Moreover, we propose to generate an
auxiliary domain by data augmentation to enforce a new domain-level metric
regularization. Experimental results on public benchmarks show the
effectiveness and accuracy of the proposed method. The code is available at
https://github.com/jinlong17/DA-Detect.

##  1 Introduction

Autonomous driving has wide applications for intelligent transportation
systems, such as improving the efficiency in the automatic 24/7 working
manner, reducing the labor costs, enhancing the comfortableness of customers,
and so on [23, 51]. With the computer vision and artificial intelligence
techniques, object detection plays a critical role in autonomous driving to
understand the surrounding driving scenarios [54, 59]. In some cases, the
autonomous vehicle might work in the complex residential and industry areas.
The diverse weather conditions might make the object detection in these
environments more difficult. For example, the usages of heating, gas, coal,
and vehicle emissions in residential and industry areas might be possible to
generate more frequent foggy or hazy weather, leading to a significant
challenge to the object detection system installed on the autonomous vehicle.

Figure 1: Illustration of the domain adaptive object detection for autonomous
driving: (a) Faster R-CNN [37] detection under clear weather, (b) Faster R-CNN
detection under foggy weather without domain adaptation, (c) Faster R-CNN
detection under foggy weather with the proposed domain adaptation.

Many deep learning models such as Faster R-CNN [37], YOLO [36] have
demonstrated great success in autonomous driving. However, most of these well-
known methods assume that the feature distributions of training and testing
data are homogeneous. Such an assumption may fail when taking the real-world
diverse weather conditions into account [40]. For example, as shown in Fig. 1,
the Faster R-CNN model trained on the clear-weather data (source domain) is
capable of detecting objects accurately under good weather, but its
performance drops significantly when it comes to the foggy weather (target
domain). This degradation is caused by the feature domain gap between
divergent weather conditions, as the model is unfamiliar with the feature
distribution on the target domain, while the detection performance could be
improved under the foggy weather with domain adaptation.

Domain adaptation, as a technique of transfer learning, is to reduce the
domain shift between various weathers. This paper proposes a novel domain
adaptation framework to achieve robust object detection performance in
autonomous driving under foggy weather. As manually annotating images under
adverse weathers is usually time-consuming, our design follows an unsupervised
fashion same as that in [5, 43, 26], where clear-weather images (source
domain) are well labeled and foggy weather images (target domain) have no
annotations. Inspired by [5, 15], our method leverages both image-level and
object-level adaptation to diminish the domain discrepancy in image style and
object appearance jointly, which is realized by involving image-level and
object-level domain classifiers to enable our convolutional neural networks
generating domain-invariant latent feature representations. Specifically, the
domain classifiers aim to maximize the probability of distinguishing the
features produced by different domains, whereas the detection model expects to
generate the domain-invariant features to confuse the classifiers.

This paper also addresses two critical insights that are ignored by previous
domain adaptation methods [5, 26, 9, 61, 15]: 1) Different training samples
might have different challenging levels to be fully harnessed during the
transfer learning, while existing works usually omit such diversity; 2)
Previous domain adaptation methods only consider the source domain and target
domain for transfer learning, while the domain-level feature metric distance
to the third related domain might be neglected. However, embedding the mining
for hard examples and involving an extra related domain might potentially
further enhance the model’s robust learning capabilities, which has not been
carefully explored before. To emphasize these two insights, we propose a new
Adversarial Gradient Reversal Layer (AdvGRL) and generate an auxiliary domain
by data augmentation. The AdvGRL performs adversarial mining for the hard
examples to enhance the model learning on the challenging scenarios, and the
auxiliary domain enforces a new domain-level metric regularization during the
transfer learning. Experimental results on the public benchmarks Cityscapes
[7] and Foggy Cityscapes [40] show the effectiveness of each proposed
component and the superior object detection performance over the baseline and
comparison methods. Overall, the contributions of this paper are summarized as
follows:

  * •

We propose a novel deep transfer learning based domain adaptive object
detection framework for autonomous driving under foggy weather, including the
image-level and object-level adaptations, which is trained with labeled clear-
weather data and unlabeled foggy-weather data to enhance the generalization
ability of the deep learning based object detection model.

  * •

We propose a new Adversarial Gradient Reversal Layer (AdvGRL) to perform
adversarial mining for the hard examples together with the domain adaptation
to further enhance the model’s transfer learning capabilities under
challenging samples.

  * •

We propose a new domain-level metric regularization during the transfer
learning. By generating an auxiliary domain with data augmentation, the
domain-level metric constraint between source domain, auxiliary domain, and
target domain is ensured as regularization during the transfer learning.

##  2 Related Work

###  2.1 Object detection for autonomous driving

Recent advancement in deep learning has brought outstanding progress in
autonomous driving [33, 6, 25, 53], and object detection has been one of the
most active topic under this field [41, 59, 8, 45]. Regarding the network
architecture, current object detection algorithms can be roughly split into
two categories: two-stage methods and single-stage methods. Two-stage object
detection algorithms typically compose of two processes: 1) region proposal,
2) object classification and localization refinement. R-CNN [14] is the first
work for this kind of methods, it applies selective search for regional
proposals and independent CNNs for each object prediction. Fast R-CNN [13]
improves R-CNN by obtaining object features from the shared feature map
learned by one CNN. Faster R-CNN [37] further enhances the framework by
proposing Region Proposal Network (RPN) to replace the selective search stage.
Single-stage object detection algorithms predict object bounding boxes and
classes simultaneously in one same stage. These methods usually leverage pre-
defined anchors to classify objects and regress bounding boxes, they are less
time-consuming but less accurate compared to two-stage algorithms. Milestones
for this category include SSD-series [29], YOLO-series [36] and RetinaNet
[28]. Despite their success in clear-weather visual scenes, these object
detection methods might not be employed in autonomous driving directly due to
the complex real-world weather conditions.

###  2.2 Object detection for autonomous driving under different weather

In order to address the diverse weather conditions encountered in autonomous
driving, many datasets have been generated [40, 31, 32, 34] and many methods
have been proposed [22, 17, 35, 2, 42, 44, 18] in recent years. For example,
Foggy Cityscape [40] is a synthetic dataset that applies fog simulation to
Cityscape for scene understanding in foggy weather. TJU-DHD [32] is a diverse
dataset for object detection in real-world scenarios which contains variances
in terms of illumination, scene, weather and season. In this paper, we focus
on the object detection problem in foggy weather. Huang et al. [22] propose a
DSNet (Dual-Subnet Network) that involves a detection subnet and a restoration
subnet. This network can be trained with multi-task learning by combining
visibility enhancement task and object detection task, thus outperforms pure
object detectors. Hahner et al. [17] develop a fog simulation approach to
enhance existing real lidar dataset, and show this approach can be leveraged
to improve current object detection methods in foggy weather. Qian et al. [35]
propose a MVDNet (Multimodal Vehicle Detection Network) that takes advantage
of lidar and radar signals to obtain proposals. Then the region-wise features
from these two sensors are fused together to get final detection results.
Bijelic et al. [2] develop a network that takes the data from four sensors as
input: lidar, RGB camera, gated camera, and radar. This architecture uses
entropy-steered adaptive deep fusion to get fused feature maps for prediction.
These methods typically rely on input data from other sensors rather than RGB
camera itself, which is not the general case for many autonomous driving cars.
Thus we aim to develop an object detection architecture that only takes RGB
camera data as input in this work.

###  2.3 Domain adaptation for object detection

Domain adaptation reduces the discrepancy between different domains, thus
allows the model trained on source domain to be applicable on unlabeled target
domain. Previous domain adaptation works mainly focus on the task of image
classification [46, 47, 56, 48], while more and more methods have been
proposed to solve domain adaptation for object detection in recent years [5,
24, 39, 60, 50, 55, 58, 49, 15]. Domain adaptive detectors could be obtained
if the features from different domains are aligned [5, 18, 39, 49, 15, 52].
From this perspective, Chen et al. [5] introduce a Domain Adaptive Faster
R-CNN framework to reduce domain gap from image level and instance level, and
the image-and-instance consistency is subsequently employed to improve cross-
domain robustness. He et al. [18] propose a MAF (multi-adversarial Faster
R-CNN) framework to minimize the domain distribution disparity by aligning
domain features and proposal features hierarchically. On the other hand, some
works try to solve domain adaptation through image style transfer methods [41,
24, 21]. Shan et al. [41] first convert images from source domain to target
domain with image translation module, then train the object detector with
adversarial training on target domain. Hsu et al. [21] choose to translate
images progressively, and add a weighted task loss during adversarial training
stage for tackling the problem of image quality difference. Many previous
methods [62, 27, 38, 4] design complex architectures. [62] used multi-scale
backbone Feature Pyramid Networks and considered pixel-level and category-
level adaptation. [27] used the complex Graph Convolution Network and graph
matching algorithms. [38] used the similarity-based clustering and grouping.
[4] uses the uncertainty-guided self-training mechanism (Probabilistic Teacher
and Focal Loss) to capture the uncertainty of unlabeled target data from a
gradually evolving teacher and guides student learning. Differently, our
method does not bring extra learnable parameters to original Faster R-CNN
model because our AdvGRL is based on adversarial training (gradient reversal)
and Domain-level Metric Regularization is based on triplet loss. Previous
domain adaptation methods usually treat training samples at the same
challenging level, while we employ advGRL for adversarial hard example mining
to improve transfer learning. Moreover, we generate an auxiliary domain and
apply domain-level metric regularization to avoid overfitting.

##  3 Proposed Method

In this section, we will first introduce the overall network architecture,
then describe the image-level and object-level adaptation method, and finally,
reveal the details of AdvGRL and domain-level metric regularization.

###  3.1 Network Architecture

Figure 2: The architecture of proposed domain adaptive object detection for
autonomous driving under foggy weather. Based on the traditional Faster R-CNN
architecture [37], the image-level and object-level domain adaptations with
adversarial gradient reversal layer (AdvGRL) and domain-level metric
regularization are designed in the proposed framework. This figure is best
viewed in color.

As illustrated in Fig. 2, our proposed model adopts the pipeline in Faster
R-CNN for object detection. The Convolutional Neural Network (CNN) backbone
extracts the image-level features from the RGB images and send them to Region
Proposal Network (RPN) to generate object proposals. Afterwards, the ROI
pooling accepts both image-level features and object proposals as the input to
retrieve the object-level features. Eventually, a detection head is applied on
the object-level features to produce the final predictions. Based on the
Faster R-CNN framework, we integrate two more components: image-level domain
adaptation module, and object-level domain adaptation module. For both
modules, we deploy a new Adversarial Gradient Reversal Layer (AdvGRL) together
with the domain classifier to extract domain-invariant features and perform
adversarial hard example mining. Moreover, we involve an auxiliary domain to
impose a new domain-level metric regularization to enforce the feature metric
distance between different domains. All three domains, i.e., source, target,
and auxiliary domains, will be employed simultaneously during the training.

###  3.2 Image-level Adaptation

The image-level domain representation is obtained from the backbone feature
extraction and contains rich global information such as style, scale and
illumination, which can potentially pose significant impacts on the detection
task [5]. Therefore, a domain classifier is introduced to classify the domains
of the upcoming image-level features to enhance the image-level global
alignment. The domain classifier is just a simple CNN with two convolutional
layers and it will output a prediction to identify the feature domain. We use
the binary cross entropy loss for the domain classifier as follows:

$L_{img}=-\sum_{i=1}^{N}[G_{i}{\rm log}P_{i}+(1-G_{i}){\rm log}(1-P_{i})],$
(1)

where $i\in\\{1,...,N\\}$ represents the $N$ training images,
$G_{i}\in\\{1,0\\}$ is the ground truth of the domain label in the $i$-th
training image ($1$ and $0$ stand for source and target domains respectively),
and $P_{i}$ is the prediction of the domain classifier.

###  3.3 Object-level Adaptation

Besides the image-level global difference in different domains, the objects in
different domains might be also dissimilar in the appearance, size, color,
etc. In this paper, we define each region proposal after the ROI Pooling layer
in Faster R-CNN as a potential object. Similar with image-level adaptation
module, after retrieving the object-level domain representation by ROI
pooling, we implement a object-level domain classifier to identify the feature
derivation from local information. A well-trained object-level classifier, a
neural network with 3 fully-connected layers, will help align the object-level
feature distribution. We also use the binary cross entropy loss for this
domain classifier:

$L_{obj}=-\sum_{i=1}^{N}\sum_{j=1}^{M}[G_{i,j}{\rm log}P_{i,j}+(1-G_{i,j}){\rm
log}(1-P_{i,j})],$ (2)

where $j\in\\{1,...,M\\}$ is the $j$-th detected object (region proposal) in
the $i$-th image, $P_{i,j}$ is the prediction of the object-level domain
classifier for the $j$-th region proposal in the $i$-th image, and $G_{i,j}$
is the corresponding binary ground-truth label for source and target domains
respectively.

###  3.4 Adversarial Gradient Reversal Layer

Figure 3: Illustration of the adversarial mining for hard training examples by
the proposed AdvGRL. In this example, we set $\lambda_{0}=1$, $\beta=30$.
Harder training examples with lower domain classifier loss $L_{c}$ will have
larger response. Figure 4: The example of generating auxiliary domain using
Cityscapes dataset: (a) an original image, (b) a rain map by RainMix [16], (c)
a synthetic rainy Cityscapes image for the auxiliary domain.

In this section, we first review the original Gradient Reversal Layer (GRL)
[10], then we make a detailed description of the proposed Adversarial Gradient
Reversal Layer (AdvGRL) for our domain adaptive object detection framework.

The original GRL is used for unsupervised domain adaptation of the image
classification task [10]. Specifically, it leaves the input unchanged during
forward propagation and reverses the gradient by multiplying it by a negative
scalar when back-propagating to the base network ahead during training. A
domain classifier is trained to maximize the probability of identifying the
domain while the base network ahead is optimized to confuse the domain
classifier. In this way, the domain-invariant features are obtained to realize
the domain adaptation. The forward propagation of GRL is defined as:

$R_{\lambda}({\mathbf{v}})={\mathbf{v}},$ (3)

where ${\mathbf{v}}$ is an input feature vector, and $R_{\lambda}$ denotes the
forwarding function that GRL performs, and the back-propagation of GRL is
defined as:

$\frac{dR_{\lambda}}{d{\mathbf{v}}}=-\lambda\mathbf{I},$ (4)

where $\mathbf{I}$ is an identity matrix and $-\lambda$ is a negative scalar.

The original GRL sets either a constant or a changing $-\lambda$ based on the
training iterations [10]. However, this setting ignores an insight that
different training samples might have different challenging levels during the
transfer learning. Therefore, this paper proposes a novel AdvGRL to perform
adversarial mining for the hard examples together with the domain adaptation
to further enhance the model’s transfer learning capabilities under
challenging examples. This can be done by simply replacing $\lambda$ by a new
$\lambda_{adv}$ in Eq. (4) of GRL, which forms the proposed AdvGRL.
Particularly, $\lambda_{adv}$ is calculated as:

$\lambda_{adv}=\left\\{\begin{aligned} &{\rm
min}(\frac{\lambda_{0}}{L_{c}},\beta),\qquad&L_{c}<\alpha\\\
&\lambda_{0},\qquad&{\rm otherwise},\end{aligned}\right.$ (5)

where $L_{c}$ is the loss of the domain classifier, $\alpha$ is a hardness
threshold to judge whether the training sample is challenging, $\beta$ is the
overflow threshold to avoid generating excessive gradients in the back-
propagation, and $\lambda_{0}=1$ is set as a fixed parameter in our
experiment. In other words, if the domain classifier’s loss $L_{c}$ is
smaller, the domain of the training sample can be more easily identified,
whose feature is not the desired domain-invariant feature, so this kind of
training sample is a harder example for domain adaptation. The relation of
$\lambda_{adv}$ and $L_{c}$ is shown in Fig. 3.

On summary, the proposed AdvGRL has two effects: 1) AdvGRL could use negative
gradients during back-propagation to confuse the domain classifier so as to
generate domain-invariant features; 2) AdvGRL could perform adversarial mining
for the hard examples to further enhance the model generalization under
challenging examples. The proposed AdvGRL is applied to both image-level and
object-level domain adaptation in our domain adaptive object detection
framework, as shown in Fig. 2.

###  3.5 Domain-level Metric Regularization

Previous existing domain adaptation methods mainly focus on the transfer
learning from source domain $S$ to target domain $T$, which neglects the
potential benefits of the third related domain can bring. To address this and
thus additionally involve the feature metric constraint between different
domains, we introduce an auxiliary domain for a domain-level metric
regularization during the transfer learning.

Based on the source domain $S$, we can apply some advanced data augmentation
methods to generate an auxiliary domain $A$. For the autonomous driving
scenario, the training data in different weather conditions can be synthesized
from the clear-weather data, then the three input images of our architecture
(as shown in Fig. 2) could be aligned images. For example, we generate an
auxiliary domain with the advanced data augmentation method RainMix [16, 20].
Specifically, we randomly sample a rain map from the public dataset of real
rain streaks [11], then perform random transformations using the RainMix
technique on the rain map, where these random transformations (i.e., rotate,
zoom, translate, shear) are sampled and combined. Finally, these transformed
rain maps can be blended with the source domain images, which can simulate the
diverse rain patterns in the real world. The example of generating auxiliary
domain is shown in Fig. 4. Different with other methods including data
augmentation to the source/target domain, by generating an auxiliary domain
with data augmentation, the domain-level metric constraint between source,
auxiliary, and target domains is ensured.

Let us define the $i$-th training image’s global image-level features of $S$,
$A$ and $T$ as $F_{i}^{S}$, $F_{i}^{A}$, and $F_{i}^{T}$ respectively. We
expect to ensure the feature metric distance between the $F_{i}^{S}$ and
$F_{i}^{T}$ closer than the feature metric distance between $F_{i}^{S}$ and
$F_{i}^{A}$ after reducing the domain gap between $S$ and $T$, which is
defined as:

$d(F_{i}^{S},F_{i}^{T})<d(F_{i}^{S},F_{i}^{A}),$ (6)

where $d(,)$ denotes the metric distance of the corresponding features. This
constraint can be implemented by a triplet structure, where the $F_{i}^{S}$,
$F_{i}^{T}$, $F_{i}^{A}$ can be treated as anchor, positive and negative in
the triplet structure. Therefore, as the domain-level metric regularization on
image features, the above image-level constraint in Eq. (6) is equivalent to
minimize the following image-level triplet loss:

$L^{R}_{img}={\rm
max}(d(F_{i}^{S},F_{i}^{T})-d(F_{i}^{S},F_{i}^{A})+\delta,0),$ (7)

where the parameter $\delta$ is used as a margin constraint and we set
$\delta=1.0$ in our experiments.

Similarly, let us define the $i$-th training image’s $j$-th object-level
features of $S$, $A$ and $T$ as $f_{i,j}^{S}$, $f_{i,j}^{A}$, and
$f_{i,j}^{T}$ respectively. As the domain-level metric regularization on
object features, we will also minimize the following object-level triplet
loss:

$L^{R}_{obj}={\rm
max}(d(f_{i,j}^{S},f_{i,j}^{T})-d(f_{i,j}^{S},f_{i,j}^{A})+\delta,0).$ (8)

###  3.6 Loss Function

The final training loss of the proposed network is a summation of each
individual part, which can be written as:

$L=L_{cls}+L_{reg}+w*(L_{img}+L_{obj}+L^{R}_{img}+L^{R}_{obj}),$ (9)

where $L_{cls}$ and $L_{reg}$ are the loss of classification and the loss of
regression in the original Faster R-CNN respectively, and $w$ is a weight to
balance the Faster R-CNN loss and the domain adaptation loss for training. We
set $w=0.1$ in our experiments. In the training, the proposed domain adaptive
object detection framework can be trained in an end-to-end manner using a
standard Stochastic Gradient Descent algorithm. During the testing, the
original Faster R-CNN architecture with trained adapted weights can be used
for object detection, after removing the domain adaptation components.

###  3.7 General Domain Adaptive Object Detection

Our model has the capability to be adapted to general domain adaptive object
detection. For the scenarios that the images from target domain are
synthesized from the source domain with pixel-to-pixel correspondence (e.g.,
Cityscapes$\longrightarrow$Foggy Cityscapes), our method can be directly
applied without modification. For the scenarios where target and source
domains do not have strict correspondence (e.g.,
Cityscapes$\longrightarrow$KITTI), our method can be applied by simply
removing the $L^{R}_{obj}$ loss to eliminate the dependence on the object
alignment in the model training.

##  4 Experiments

###  4.1 Benchmark

Our experiments are based on the public object detection benchmarks Cityscapes
[7] and Foggy Cityscapes [40] for autonomous driving. Cityscapes [7] is a
widely used autonomous driving dataset, which is a collection of images with
city street scenarios in clear weather conditions from 27 cities. In
Cityscapes dataset, there are 2,975 training images and 500 validation images
with instance segmentation annotations which can be transformed into bounding-
box annotations with $8$ categories. All images are 3-channel RGB images and
captured by a car-mounted video camera with the same resolution of $1024\times
2048$. Foggy Cityscapes [40] is established by simulating the fog of different
intensity levels on the Cityscapes images, which generates the simulated three
levels of fog based on the depth map and a physical model [40]. Its image
number, resolution, training/validation split, and annotations are same as
those of Cityscapes. Following the previous methods [5, 49, 15], the images
with the fog of highest intensity level are utilized as the target domain for
transfer learning in our experiments.

###  4.2 Experimental Setting

Table 1: AP for each class and overall mAP with comparison methods on the Cityscapes$\longrightarrow$Foggy Cityscapes experiment (%) as clear to foggy adaptation. Note that the best performance is bold and the second best is underlined. Methods | bus | bicycle | car | mcycle | person | rider | train | truck | mAP
---|---|---|---|---|---|---|---|---|---
SCDA-CVPR’19 [64] | 39.0 | 33.6 | 48.5 | 28.0 | 33.5 | 38.0 | 23.3 | 26.5 | 33.8
DM-CVPR’19 [24] | 38.4 | 32.2 | 44.3 | 28.4 | 30.8 | 40.5 | 34.5 | 27.2 | 34.6
MAF-ICCV’19 [18] | 39.9 | 33.9 | 43.9 | 29.2 | 28.2 | 39.5 | 33.3 | 23.8 | 34.0
MCAR-ECCV’20 [60] | 44.1 | 36.6 | 43.9 | 37.4 | 32.0 | 42.1 | 43.4 | 31.3 | 38.8
SWDA-CVPR’19 [39] | 36.2 | 35.3 | 43.5 | 30.0 | 29.9 | 42.3 | 32.6 | 24.5 | 34.3
PDA-WACV’20 [21] | 44.1 | 35.9 | 54.4 | 29.1 | 36.0 | 45.5 | 25.8 | 24.3 | 36.9
MTOR-CVPR-19 [3] | 38.6 | 35.6 | 44.0 | 28.3 | 30.6 | 41.4 | 40.6 | 21.9 | 35.1
DA-Faster-CVPR’18 [5] | 49.8 | 39.0 | 53.0 | 28.9 | 35.7 | 45.2 | 45.4 | 30.9 | 41.0
GPA-CVPR’20 [49] | 45.7 | 38.7 | 54.1 | 32.4 | 32.9 | 46.7 | 41.1 | 24.7 | 39.5
RPN-PR-CVPR’21 [58] | 43.6 | 36.8 | 50.5 | 29.7 | 33.3 | 45.6 | 42.0 | 30.4 | 39.0
UaDAN-TMM’21 [15] | 49.4 | 38.9 | 53.6 | 32.3 | 36.5 | 46.1 | 42.7 | 28.9 | 41.1
Ours w/o Auxiliary Domain | 48.4 | 36.7 | 53.5 | 26.1 | 36.1 | 45.9 | 39.1 | 29.3 | 40.2
Ours | 51.2 | 39.1 | 54.3 | 31.6 | 36.5 | 46.7 | 48.7 | 30.3 | 42.3
Oracle | 49.9 | 45.8 | 65.2 | 39.6 | 46.5 | 51.3 | 34.2 | 32.6 | 45.6

Dataset setting: We set the labeled training set of Cityscapes [7] as source
domain and the unlabeled training set of Foggy Cityscapes [40] as target
domain during the training. Then, the trained model is tested on the
validation set of Foggy Cityscapes to report the evaluation result. We denote
this setting as the Cityscapes$\longrightarrow$Foggy Cityscapes experiment in
this paper.

Training and parameter setting: In the experiments, we adopt ResNet-50 as the
backbone for the Faster R-CNN [37] detection network, which is pre-trained on
ImageNet. During training, following setting in [5, 37], the back-propagation
and stochastic gradient descent (SGD) are used to optimize all the networks.
The whole network is trained with an initial learning rate $0.01$ for $50k$
iterations and then reduced to $0.001$ for another $20k$ iterations. For all
experiments, a weight decay of $0.0005$ and a momentum of $0.9$ are used, and
each batch includes an image of source domain, an image of target domain and
an image of auxiliary domain. For comparison, the $\lambda$ in the original
GRL (Eq. (4)) is set as 1. The hardness threshold $\alpha$ in the AdvGRL (Eq.
(5)) is set as $0.63$ by averaging the values of Eq. (1) when
$P_{i}=0.7,G_{i}=1$ and $P_{i}=0.3,G_{i}=0$. Our code is implemented with
PyTorch and Mask R-CNN Benchmark Toolbox [30], and all models are trained
using a GeForce RTX3090 GPU card with 24GB memory.

Evaluation metrics and comparison methods: We set the Intersection over Union
(IoU) threshold as $0.5$ to compute the Average Precision (AP) of each
category and mean Average Precision (mAP) of all categories. Then we compare
our proposed method with some recent domain adaptation comparison methods in
our experiments, such as SCDA [64], DM [24], MAF [18], MCAR [60], SWDA [39],
PDA [21], RPN-PR [58], MTOR [3], DA-Faster [5], GPA [49], and UaDAN [15].

###  4.3 Clear to Foggy Adaptation

The results of weather adaptation from clear weather to foggy weather are
represented in Table 1. Compared with other domain adaptation methods, we can
see that our proposed method achieves the best detection performance with a
mAP of $42.3\%$, which is higher than the second best method UaDAN [15] with a
mAP improvement of $1.2\%$. For each category, we can see that the proposed
method is able to alleviate the domain gap over most of the categories in
Foggy Cityscapes, e.g., bus got $51.2\%$, bicycle got $39.1\%$, person got
$36.5\%$, rider got $46.7\%$, and train got $48.7\%$ as the best performance
in AP, which is highlight in Table 1. The proposed method can reach the
$48.7\%$ AP for the train detection in Foggy Cityscapes, compared to the
$45.4\%$ AP by the second best method DA-Faster, where the proposed method is
$3.3\%$ better than DA-Faster. While PDA got $54.4\%$ in car, GPA got $32.4\%$
in motorcycle, DA-faster got $30.9\%$ in truck as the best performance in some
categories, the proposed method is comparable across these three categories
with a minor difference. Obviously, compared to these recent domain adaptation
methods, the proposed method achieves the best performance in overall mAP
performance and more than half categories of Foggy Cityscapes.

Table 2: Ablation study for mAP on the Cityscapes$\longrightarrow$Foggy Cityscapes experiment. | img | obj | AdvGRL | Reg | mAP
---|---|---|---|---|---
Source only |  |  |  |  | 23.41
img+GRL | ✓ |  |  |  | 38.10
obj+GRL |  | ✓ |  |  | 38.02
img+obj+GRL (Baseline) | ✓ | ✓ |  |  | 38.43
img+obj+AdvGRL | ✓ | ✓ | ✓ |  | 40.23
img+obj+GRL+Reg | ✓ | ✓ |  | ✓ | 41.97
img+obj+AdvGRL+Reg | ✓ | ✓ | ✓ | ✓ | 42.34

###  4.4 Cross-Camera Adaptation

To fully evaluate the proposed method, we conduct an experiment to perform the
cross-camera adaptation between real-world autonomous driving datasets with
different camera settings. To apply our method to the unaligned datasets in
the real-world, we simply remove $L^{R}_{obj}$ (Eq. 8) to apply our method
from Cityscapes (source) to KITTI [12] (target) datasets for cross-camera
adaptation. Same as [5], we use KITTI training set (7,481 images of resolution
$1250\times 375$) as target domain in both adaptation and evaluation, and AP
of Car on target domain is evaluated. The result is in Table 3, where the
proposed method achieved outstanding performance compared with recent SOTA
methods.

Table 3: AP of Car on the Cityscapes$\longrightarrow$KITTI experiment as cross-camera adaptation. |  MAF-ICCV’19[18] |  ATF-ECCV’20[19] |  ART-CVPR’20[61] |  GPA-CVPR’20[49] |  SGA-TMM’21[57] |  UIT-ESwA’22[1] | Ours
---|---|---|---|---|---|---|---
AP | 72.10 | 73.50 | 73.60 | 65.36 | 72.02 | 73.70 | 74.38

###  4.5 Ablation Study on Components

The effect of each individual proposed component for the domain adaptation
detection method is investigated in this section. All experiments are
conducted with the same RestNet-50 backbone on the
Cityscapes$\longrightarrow$Foggy Cityscapes experiment. The results are
presented in Table 2. In the first row, ‘img’ and ‘obj’ stand for the image-
level adaptation module and object-level adaptation module respectively, while
‘AdvGRL’ and ‘Reg’ denote the proposed Adversarial Gradient Reversal Layer and
domain-level metric Regularization respectively. ‘img+obj+GRL’ stands for the
baseline model in our experiment. We denote that ‘img+obj+AdvGRL’ (Ours w/o
Auxiliary Domain) and ‘img+obj +AdvGRL+Reg’ use the AdvGRL to replace the
original GRL. The ‘Source only’ indicates the Faster R-CNN model without
domain adaptation trained only with labeled source domain images. The ablation
study in Table 2 clearly justifies the positive effect of each proposed
component of the domain adaptive object detection.

###  4.6 Ablation Study on Parameters

The study on different hyper-parameters of Eq. 9 and Eq. 5 are conducted. We
use the Cityscapes$\longrightarrow$Foggy Cityscapes as the study case. First,
the loss balance weight $w$ in Eq. 9 is set as $0.1$, $0.01$, $0.001$
separately for training, and the corresponding detection mAP(s) are 42.34,
41.30, 41.19, respectively. Second, in the AdvGRL (Eq. 5), the overflow
threshold $\beta$ and hardness threshold $\alpha$ are set as (1)
$\beta=30,\alpha=0.63$, (2) $\beta=10,\alpha=0.63$, (3)
$\beta=30,\alpha=0.54$, and (4) $\beta=10,\alpha=0.54$, where $\alpha=0.54$ is
computed by averaging the values of Eq. 1 when $P_{i}=0.9,G_{i}=1$ and
$P_{i}=0.1,G_{i}=0$. The detection mAP(s) of these settings are (1) 42.34, (2)
38.83, (3) 39.38, (4) 40.47, respectively.

###  4.7 Discussion on Visualized Hard Examples

Using $\lambda_{adv}$ of the proposed AdvGRL, we could find some hard
examples, as shown in Fig. 5. We compute the $L_{1}$ distance of features
$F^{S}_{i}$ and $F^{T}_{i}$ after the CNN backbone of Fig. 2 as the
approximated hardness $ah$, where smaller $ah$ means harder for transfer
learning. Intuitively, if the fog covers more objects as shown in bounding-box
regions of Fig. 5, it will be more difficult.

Figure 5: Hard examples (bigger $\lambda_{adv}$) mined by AdvGRL. Left to
right: two mined hard examples, an easy example.

###  4.8 Discussion on Domain Randomization, Pre-trained Models, and
Qualitative Results

Domain Randomization: Domain randomization might be used to reduce the domain
shift between source domain and target domain. We use two ways as the domain
randomization in the Cityscapes$\longrightarrow$Foggy Cityscapes experiment,
i.e., regular data argumentation and CycleGAN [63] based image style transfer.
1) We construct the auxiliary domain by regular data argumentation (color
change + blur + salt & pepper noises), where our method’s detection mAP is
38.7, compared to our 42.3 by the auxiliary domain with rain synthesis. 2) We
train a CycleGAN to transfer the image style between the training sets of
Cityscapes and Foggy Cityscapes. Using the generated fake foggy-style image of
Cityscapes by the trained CycleGAN model to train a Faster R-CNN model, it
achieves detection mAP as 32.8. These experiments show that commonly used
domain randomization could not well solve the domain adaptation problem.

Pre-trained Models: We use the pre-trained Faster R-CNN model in [5] to
initialize our method, then our method gets the detection mAP as 41.3 in the
Cityscapes$\longrightarrow$Foggy Cityscapes experiment, compared to 42.3 by
our method without pre-trained detection model.

Qualitative Results: We visualize some detection results on the Foggy
Cityscapes dataset in Fig. 6, which shows that the proposed domain adaptive
method improves the detection performance in foggy weather significantly.

Figure 6: Qualitative visualization results on validation set of Foggy
Cityscapes: (a) Original Faster R-CNN without domain adaptation, (b) Faster
R-CNN with image-level and object-level adaptations using GRL (Baseline), (c)
Proposed Method. Note: different colors represent different categories.

##  5 Conclusions

In this paper, we propose a novel domain adaptive object detection framework
for autonomous driving. The image-level and object-level adaptations are
designed to reduce the domain shift on the global image style and local object
appearance. A new adversarial gradient reversal layer is proposed to perform
adversarial mining for hard examples together with domain adaptation.
Considering the feature metric distance between the source domain, target
domain, and auxiliary domain by data augmentation, we propose a new domain-
level metric regularization. Furthermore, our method could be applied to solve
the general domain adaptive object detection problem. We conduct the transfer
learning experiments from Cityscapes to Foggy Cityscapes and from Cityscapes
to KITTI, and experimental results show that the proposed method is quite
effective.

Acknowledgement: This work was supported by NSF 2215388.

## References

  * [1] Vinicius F Arruda, Rodrigo F Berriel, Thiago M Paixão, Claudine Badue, Alberto F De Souza, Nicu Sebe, and Thiago Oliveira-Santos.  Cross-domain object detection using unsupervised image translation.  Expert Systems with Applications, 192:116334, 2022.
  * [2] Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, and Felix Heide.  Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 11682–11692, 2020.
  * [3] Qi Cai, Yingwei Pan, Chong-Wah Ngo, Xinmei Tian, Lingyu Duan, and Ting Yao.  Exploring object relation in mean teacher for cross-domain detection.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 11457–11466, 2019.
  * [4] Meilin Chen, Weijie Chen, Shicai Yang, Jie Song, Xinchao Wang, Lei Zhang, Yunfeng Yan, Donglian Qi, Yueting Zhuang, Di Xie, et al.  Learning domain adaptive object detection with probabilistic teacher.  In International Conference on Machine Learning, pages 3040–3055. PMLR, 2022.
  * [5] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Domain adaptive faster r-cnn for object detection in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3339–3348, 2018.
  * [6] Henrik Christensen, David Paz, Hengyuan Zhang, Dominique Meyer, Hao Xiang, Yunhai Han, Yuhan Liu, Andrew Liang, Zheng Zhong, and Shiqi Tang.  Autonomous vehicles for micro-mobility.  Autonomous Intelligent Systems, 1(1):1–35, 2021.
  * [7] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3213–3223, 2016.
  * [8] Di Feng, Ali Harakeh, Steven L Waslander, and Klaus Dietmayer.  A review and comparative study on probabilistic object detection in autonomous driving.  IEEE Transactions on Intelligent Transportation Systems, 2021.
  * [9] Lan Fu, Hongkai Yu, Felix Juefei-Xu, Jinlong Li, Qing Guo, and Song Wang.  Let there be light: Improved traffic surveillance via detail preserving night-to-day transfer.  IEEE Transactions on Circuits and Systems for Video Technology, 2021\.
  * [10] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In International Conference on Machine Learning, pages 1180–1189. PMLR, 2015.
  * [11] Kshitiz Garg and Shree K Nayar.  Photorealistic rendering of rain streaks.  ACM Transactions on Graphics, 25(3):996–1002, 2006.
  * [12] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In 2012 IEEE conference on computer vision and pattern recognition, pages 3354–3361. IEEE, 2012.
  * [13] Ross Girshick.  Fast r-cnn.  In IEEE International Conference on Computer Vision, pages 1440–1448, 2015.
  * [14] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 580–587, 2014.
  * [15] Dayan Guan, Jiaxing Huang, Aoran Xiao, Shijian Lu, and Yanpeng Cao.  Uncertainty-aware unsupervised domain adaptation in object detection.  IEEE Transactions on Multimedia, 2021.
  * [16] Qing Guo, Jingyang Sun, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Wei Feng, Yang Liu, and Jianjun Zhao.  Efficientderain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining.  In AAAI Conference on Artificial Intelligence, pages 1487–1495, 2021.
  * [17] Martin Hahner, Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Fog simulation on real lidar point clouds for 3d object detection in adverse weather.  In IEEE International Conference on Computer Vision, pages 15283–15292, 2021.
  * [18] Zhenwei He and Lei Zhang.  Multi-adversarial faster-rcnn for unrestricted object detection.  In IEEE International Conference on Computer Vision, pages 6668–6677, 2019.
  * [19] Zhenwei He and Lei Zhang.  Domain adaptive object detection via asymmetric tri-way faster-rcnn.  In European conference on computer vision, pages 309–324. Springer, 2020.
  * [20] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  AugMix: A simple data processing method to improve robustness and uncertainty.  International Conference on Learning Representations, 2020.
  * [21] Han-Kai Hsu, Chun-Han Yao, Yi-Hsuan Tsai, Wei-Chih Hung, Hung-Yu Tseng, Maneesh Singh, and Ming-Hsuan Yang.  Progressive domain adaptation for object detection.  In IEEE Winter Conference on Applications of Computer Vision, pages 749–757, 2020.
  * [22] Shih-Chia Huang, Trung-Hieu Le, and Da-Wei Jaw.  Dsnet: Joint semantic learning for object detection in inclement weather conditions.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(8):2623–2633, 2020.
  * [23] Kichun Jo, Junsoo Kim, Dongchul Kim, Chulhoon Jang, and Myoungho Sunwoo.  Development of autonomous car—part i: Distributed system architecture and development process.  IEEE Transactions on Industrial Electronics, 61(12):7131–7140, 2014\.
  * [24] Taekyung Kim, Minki Jeong, Seunghyeon Kim, Seokeon Choi, and Changick Kim.  Diversify and match: A domain adaptive representation learning paradigm for object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 12456–12465, 2019.
  * [25] E Li, Shuaijun Wang, Chengyang Li, Dachuan Li, Xiangbin Wu, and Qi Hao.  Sustech points: A portable 3d point cloud interactive annotation platform system.  In IEEE Intelligent Vehicles Symposium, pages 1108–1115, 2020.
  * [26] Jinlong Li, Zhigang Xu, Lan Fu, Xuesong Zhou, and Hongkai Yu.  Domain adaptation from daytime to nighttime: A situation-sensitive vehicle detection and traffic flow parameter estimation framework.  Transportation Research Part C: Emerging Technologies, 2021.
  * [27] Wuyang Li, Xinyu Liu, and Yixuan Yuan.  Sigma: Semantic-complete graph matching for domain adaptive object detection.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5291–5300, 2022.
  * [28] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In IEEE International Conference on Computer Vision, pages 2980–2988, 2017.
  * [29] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In European Conference on Computer Vision, pages 21–37. Springer, 2016.
  * [30] Francisco Massa and Ross Girshick.  maskrcnn-benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch.  https://github.com/facebookresearch/maskrcnn-benchmark, 2018.
  * [31] Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S Ecker, Matthias Bethge, and Wieland Brendel.  Benchmarking robustness in object detection: Autonomous driving when winter is coming.  arXiv preprint arXiv:1907.07484, 2019.
  * [32] Yanwei Pang, Jiale Cao, Yazhao Li, Jin Xie, Hanqing Sun, and Jinfeng Gong.  Tju-dhd: A diverse high-resolution dataset for object detection.  IEEE Transactions on Image Processing, 30:207–219, 2020.
  * [33] David Paz, Hengyuan Zhang, Qinru Li, Hao Xiang, and Henrik I Christensen.  Probabilistic semantic mapping for urban autonomous driving applications.  In IEEE International Conference on Intelligent Robots and Systems, pages 2059–2064. IEEE, 2020.
  * [34] Quang-Hieu Pham, Pierre Sevestre, Ramanpreet Singh Pahwa, Huijing Zhan, Chun Ho Pang, Yuda Chen, Armin Mustafa, Vijay Chandrasekhar, and Jie Lin.  A* 3d dataset: Towards autonomous driving in challenging environments.  In IEEE International Conference on Robotics and Automation, pages 2267–2273. IEEE, 2020.
  * [35] Kun Qian, Shilin Zhu, Xinyu Zhang, and Li Erran Li.  Robust multimodal vehicle detection in foggy weather using complementary lidar and radar signals.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 444–453, 2021.
  * [36] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 779–788, 2016.
  * [37] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in Neural Information Processing Systems, 28, 2015.
  * [38] Farzaneh Rezaeianaran, Rakshith Shetty, Rahaf Aljundi, Daniel Olmeda Reino, Shanshan Zhang, and Bernt Schiele.  Seeking similarities over differences: Similarity-based domain alignment for adaptive object detection.  In IEEE/CVF International Conference on Computer Vision, pages 9204–9213, 2021.
  * [39] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko.  Strong-weak distribution alignment for adaptive object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 6956–6965, 2019.
  * [40] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Semantic foggy scene understanding with synthetic data.  International Journal of Computer Vision, 126(9):973–992, 2018\.
  * [41] Yuhu Shan, Wen Feng Lu, and Chee Meng Chew.  Pixel and feature level based domain adaptation for object detection in autonomous driving.  Neurocomputing, 367:31–38, 2019.
  * [42] Vishwanath A Sindagi, Poojan Oza, Rajeev Yasarla, and Vishal M Patel.  Prior-based domain adaptive object detection for hazy and rainy conditions.  In European Conference on Computer Vision, pages 763–780. Springer, 2020.
  * [43] Shaoyue Song, Hongkai Yu, Zhenjiang Miao, Jianwu Fang, Kang Zheng, Cong Ma, and Song Wang.  Multi-spectral salient object detection by adversarial domain adaptation.  In AAAI Conference on Artificial Intelligence, pages 12023–12030, 2020.
  * [44] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxim: Multi-axis mlp for image processing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5769–5780, 2022.
  * [45] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxvit: Multi-axis vision transformer.  arXiv preprint arXiv:2204.01697, 2022.
  * [46] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 7167–7176, 2017.
  * [47] Mei Wang and Weihong Deng.  Deep visual domain adaptation: A survey.  Neurocomputing, 312:135–153, 2018.
  * [48] Ni Xiao and Lei Zhang.  Dynamic weighted learning for unsupervised domain adaptation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 15242–15251, 2021.
  * [49] Minghao Xu, Hang Wang, Bingbing Ni, Qi Tian, and Wenjun Zhang.  Cross-domain detection via graph-induced prototype alignment.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 12355–12364, 2020.
  * [50] Qiangeng Xu, Yin Zhou, Weiyue Wang, Charles R Qi, and Dragomir Anguelov.  Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation.  In IEEE International Conference on Computer Vision, pages 15446–15456, 2021.
  * [51] Runsheng Xu, Yi Guo, Xu Han, Xin Xia, Hao Xiang, and Jiaqi Ma.  Opencda: an open cooperative driving automation framework integrated with co-simulation.  In IEEE International Intelligent Transportation Systems Conference, pages 1155–1162. IEEE, 2021.
  * [52] Runsheng Xu, Jinlong Li, Xiaoyu Dong, Hongkai Yu, and Jiaqi Ma.  Bridging the domain gap for multi-agent perception.  arXiv preprint arXiv:2210.08451, 2022.
  * [53] Runsheng Xu, Faezeh Tafazzoli, Li Zhang, Timo Rehfeld, Gunther Krehl, and Arunava Seal.  Holistic grid fusion based stop line estimation.  In International Conference on Pattern Recognition, pages 8400–8407. IEEE, 2021.
  * [54] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma.  OPV2V: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication.  In IEEE International Conference on Robotics and Automation, 2022\.
  * [55] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi.  St3d: Self-training for unsupervised domain adaptation on 3d object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 10368–10378, 2021.
  * [56] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Universal domain adaptation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 2720–2729, 2019.
  * [57] Chong Zhang, Zongxian Li, Jingjing Liu, Peixi Peng, Qixiang Ye, Shijian Lu, Tiejun Huang, and Yonghong Tian.  Self-guided adaptation: Progressive representation alignment for domain adaptive object detection.  IEEE Transactions on Multimedia, 24:2246–2258, 2021.
  * [58] Yixin Zhang, Zilei Wang, and Yushi Mao.  Rpn prototype alignment for domain adaptive object detector.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 12425–12434, 2021.
  * [59] Xiangmo Zhao, Pengpeng Sun, Zhigang Xu, Haigen Min, and Hongkai Yu.  Fusion of 3d lidar and camera data for object detection in autonomous vehicle applications.  IEEE Sensors Journal, 20(9):4901–4913, 2020.
  * [60] Zhen Zhao, Yuhong Guo, Haifeng Shen, and Jieping Ye.  Adaptive object detection with dual multi-label prediction.  In European Conference on Computer Vision, pages 54–69. Springer, 2020.
  * [61] Yangtao Zheng, Di Huang, Songtao Liu, and Yunhong Wang.  Cross-domain object detection through coarse-to-fine feature adaptation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 13766–13775, 2020.
  * [62] Wenzhang Zhou, Dawei Du, Libo Zhang, Tiejian Luo, and Yanjun Wu.  Multi-granularity alignment domain adaptation for object detection.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9581–9590, 2022.
  * [63] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In IEEE International Conference on Computer Vision, pages 2223–2232, 2017.
  * [64] Xinge Zhu, Jiangmiao Pang, Ceyuan Yang, Jianping Shi, and Dahua Lin.  Adapting object detectors via selective cross-domain alignment.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 687–696, 2019.


2022

[1]Vincent Milimo Masilokwa Punabantu

[1]Mechanical Engineering, University of Cape Town, Cape Town, 7700, Western
Cape, South Africa

2]Electrical Engineering, University of Cape Town, Cape Town, 7700, Western
Cape, South Africa

3]The Children’s Heart Disease Research Unit, Red Cross Children’s Hospital,
Cape Town, 7700, Western Cape, South Africa

4]Extramural Research and Internal Portfolio, South African Medical Research
Council, South Africa

# Machine Learning based Extraction of Boundary Conditions from Doppler Echo
Images for Patient Specific Coarctation of the Aorta: Computational Fluid
Dynamics Study

<EMAIL_ADDRESS>Malebogo Ngoepe<EMAIL_ADDRESS>Amit Kumar
Mishra<EMAIL_ADDRESS>Thomas Aldersley<EMAIL_ADDRESS>John
Lawrenson<EMAIL_ADDRESS>Liesl Zuhlke<EMAIL_ADDRESS>*  [
[  [

###### Abstract

Purpose- Coarctation of the Aorta (CoA) patient-specific computational fluid
dynamics (CFD) studies in resource constrained settings are limited by the
available imaging modalities for geometry and velocity data acquisition.
Doppler echocardiography has been seen as a suitable velocity acquisition
modality due to its higher availability and safety. This study aimed to
investigate the application of classical machine learning (ML) methods to
create an adequate and robust approach for obtaining boundary conditions (BCs)
from Doppler Echocardiography images, for haemodynamic modeling using CFD.

Methods- Our proposed approach combines ML and CFD to model haemodynamic flow
within the region of interest. With the key feature of the approach being the
use of ML models to calibrate the inlet and outlet boundary conditions (BCs)
of the CFD model. The key input variable for the ML model was the patients
heart rate as this was the parameter that varied in time across the measured
vessels within the study. ANSYS Fluent was used for the CFD component of the
study whilst the scikit-learn python library was used for the ML component.

Results- We validated our approach against a real clinical case of severe CoA
before intervention. The maximum coarctation velocity of our simulations were
compared to the measured maximum coarctation velocity obtained from the
patient whose geometry is used within the study. Of the 5 ML models used to
obtain BCs the top model was within 5% of the measured maximum coarctation
velocity.

Conclusion- The framework demonstrated that it was capable of taking
variations of the patients heart rate between measurements into account. Thus,
enabling the calculation of BCs that were physiologically realistic when the
heart rate was scaled across each vessel whilst providing a reasonably
accurate solution.

###### keywords:

Machine Learning, Computational Fluid Dynamics, Coarctation of the Aorta,
Boundary Conditions

##  1 Introduction

Congenital heart disease (CHD) is the most common birth defect, with a global
prevalence of approximately 9 per 1000 births. Coarctation of the aorta (CoA)
is one of the more common forms of CHD constituting approximately 7% of all
CHD’s Liu2019CHD . CoA is defined as the narrowing of a point within the
aorta, usually at the isthmus, which may be discrete or elongated and occurs
on a spectrum with varying severity. This narrowing leads to increased
resistance across the vessel thus causing upper body hypertension and reduced
blood supply to downstream vessels and body parts. The resultant pressure
difference across the coarctation is used by clinicians as a means of
diagnosis, and is referred to as the pressure gradient. A peak systolic
pressure difference of greater than 20 mmHg or diagnosis via imaging following
hypertension, warrants intervention zuhlke2019congenital ; obaid2019computer .
Common treatments for CoA are balloon angioplasty, resection with end-to-end
anastomosis (REEA) and stenting in older patients zuhlke2019congenital . The
overall objective of treatment is to alleviate the pressure difference and
restore normal flow across the coarctation. However, recoarctation may occur
post intervention and the haemodynamic environment post-surgery is a strong
driver of this occurrence cecchi2011recoarctation . Computational fluid
dynamics (CFD) has been identified as a potential solution for studying
patient haemodynamics and identifying key flow metrics such as pressure,
velocity, and wall shear stress, with high temporal (time) and spatial
resolution swanson2020patient . Another key benefit is the non-invasive nature
of the technique. Furthermore, with the design of patient specific CFD
pipelines, flow modelling can be specific to an individual patient. However,
incorporating CFD within clinical workflows has been limited due to challenges
in patient data acquisition for modelling and the skill set required to
conduct the modelling capelli2018patient . Thus, most applications of the
technique have mainly resided in research.

In resource constrained settings the design of patient-specific CFD pipelines
that are clinically applicable requires careful consideration. Patient-
specific pipelines by their nature require data from each individual patient.
The nature and complexity of the model therefore depends on data availability
and acquisition, via a specific imaging modality.

In their 2019 study, Swanson et al swanson2020patient  conducted a patient
specific CoA CFD study that used CTA and doppler transthoracic
echocardiography (Doppler TTE) as the geometry and velocity data acquisition
modalities, respectively, and an open source CFD solver swanson2020patient .
The goal of the study was to develop a pipeline that could be implemented in
resource constrained settings swanson2020patient . In addition, they
demonstrated the suitability of doppler echocardiography as a velocity data
acquisition tool and motivated the case for further study of its
implementation. This, despite current preference for phase contrast magnetic
resonance imaging (PC-MRI) in patient-specific CFD literature pc_mri_1 ;
pc_mri_2 ; pc_mri_3 ; pc_mri_4 ; pc_mri_5 . Although PC-MRI is an adequate
modality from a technical standpoint, the devices tend to be costly.
Furthermore, its implementation within the paediatric population is
challenging, as it may require that the patients are sedated, intubated and
ventilated for the procedure, to enable breathe holding, which carries risks.
Most relevant to this paper, in Swanson et al’s study a doppler
echocardiography data processing study was conducted, that highlighted the
current challenges faced when using this imaging modality. One of the main
concerns were the variations within the patient heart rate across
measurements, which is an inherent limitation of the modality. For
measurements across each vessel must be taken separately in time. Therefore,
the velocity profile at the inlet and outlets did not have the same heart
rate. To account for this, the assumption was made that changes to the
velocity profile with respect to the heart rate across each vessel were
negligible. Thus, the velocity profiles across each vessel were scaled to the
same heart rate of 120BPM. This introduced errors.

Machine learning, is a family of methods that are used to automatically
discover patterns within data bishop2016pattern . They have proved effective
function approximators in cases where an analytical expression is difficult to
obtain but data is available. In CFD, ML is typically used to decrease the
high computational cost associated with traditional numerical flow solution
calculations feiger2020accelerating ; nita2022personalized ;
yevtushenko2021deep . This has varied between solely training an ML model to
compute flow parameters to having a hybrid approach where they are combined
with numerical methods. The general method to incorporating ML has been to
generate data using numerical simulations and training the ML model on this
data. Yevtushenko et al yevtushenko2021deep  demonstrated how Deep Learning
could be used to develop a centre-aggregated aortic haemodynamics model, to
compute parameters such as pressure gradient in CoA. They consider their
approach as an alternative to numerical modelling in cases where low
computational power and time are of high importance. While their approach uses
ML as the sole method for computing flow, Nita et al nita2022personalized  in
their study take a hybrid approach in which they embed a ML model to compute
pressure gradients in CoA within their proposed reduced order multi-scale
model. Another subclass of ML methods worth noting are physics based ML
methods, where physical laws such as mass continuity and the Navier-Stokes
equations are embedded into the structure of the model. Raissi et al
raissi2019pinn  for example, introduce physics informed neural networks and
demonstrate their ability to solve forward and inverse problems involving
nonlinear partial differential equations. Therefore, these studies illustrate
how ML methods can be applied to physics based problems but more so those
related to modelling flow in CoA.

The main focus of this paper is to develop a framework that uses a ML driven
approach to account for changes in the velocity profile, due to changes in
heart rate, to allow for the calculation of more accurate inlet and outlet BCs
for CFD modelling. Studies have shown the need for accurate BCs given their
effect on CFD modelling results madhavan2018effect ; morbiducci2013inflow ;
pirola2017choice . A ML approach using non Artificial Neural Network (ANN)
methods was selected as this family of models are good at finding patterns or
relationships within data. Furthermore, the size of the dataset was relatively
small, thus non ANN methods would be suitable for the problem hands-on .

The novelty of our approach is that it provides a means of determining BCs
that are physiologically realistic across all vessels in cases where doppler
echocardiography is applicable as the velocity data acquisition modality. This
contributes to the body of knowledge aimed at improving the feasibility of
doppler echocardiography as an acquisition modality in patient-specific CFD
pipelines.

This paper is arranged as follows, in the ML-CFD pipeline development section,
we describe how the full ML-CFD patient-pipeline was developed. This details
how the type of data used and how it was processed to build the dataset that
was used to train the ML models. The ML subsection describes model training
and the subsequent CFD section focuses on the modelling approach and
validation process. The Results section presents findings from the
corresponding subsections in the ML-CFD pipeline development section. The
paper then eras with the discussion and conclusion section.

##  2 Methods: ML-CFD Pipeline Development

Within this methods section we outline the methods used to develop our
approach. Figure 1 provides an overview of the full pipeline. Starting at the
data pre-procssing pipeline, key features are extracted from the provided
Doppler TTE images to build the ML training dataset. This dataset is split
into the respective inputs and outputs as shown in the ML Model Training box,
which then undergo a feature engineering process (pre-processes train input
and output in a format that can be taken in by model). Once the regression
model is trained the evaluation inputs are specified and BCs obtained. These
are then input to the CFD solver along with meshed patient geometry. The CFD
solver then outputs the velocity and pressure results.

Figure 1:  Flow Diagram representation of the full framework used to develop
the coupled ML-CFD Pipeline, for modelling patient-specific flow using Doppler
TTE as the velocity data acquisition modality

###  2.1 Data Pipeline and Clinical Data

In this study Doppler TTE images for both pre- and post- intervention across
the ascending aorta, supra aortic branches (innominate artery, left common
carotid artery and left subclavian artery), coarctation and descending aorta,
and a 3D patient-specific geometry were used, as obtained from
swanson2020patient . In addition, four images across the coarctation
constituting 1 pre- and 3 post- intervention cases were provided by Red Cross
War Memorial Children’s Hospital (RCH).

A data pre-processing pipeline was developed to process the Doppler TTE images
and build a dataset that could be used to train the ML models.

The velocities were extracted from the Doppler TTE images using a free online
plot digitizer plotdigitizer . This was a manual process conducted by the
researcher. Points were first set along the axes of the velocity graph to
convert the image to a plot.The values at the designated points were then
entered which then scaled the plot accordingly. Points were then traced along
the velocity profile within the graph to obtain the velocity readings. The
tracings were done for velocity profiles that were adequately developed, that
is, the distinct shape of the profile was visible and there was minimal noise
across the profile. Note tracings during diastole were set to zero. The
coordinates of the tracings were then saved within a csv file. This process
was repeated for all the Doppler TTE images in the dataset from
swanson2020patient  and for the 4 additional coarctation images from RCH.

Similarly the same plot digitizer was used to determine the number of beats
per second such that the heart rate of each profile could be determined. The
clinicians noted that measuring the heart rate directly from the image was a
more reliable approach. Thus, in images where the electrocardiogram (ECG) was
present, the heart rate was calculated by measuring the time interval between
successive peaks of the QRS complex. However, in images where the ECG was not
present, the time interval between the successive peaks of the velocity
profiles was used to calculate the heart rate. In both cases, the time
intervals between 4-5 successive peaks were measured and the average
calculated, which was then used as the associated heart rate. We note that
this approach introduces human error, especially in the cases where the ECG
was not present. Given that the sampling rate at which the velocity profiles
were measured was not available methods such as using a Fourier transform were
not an option. As such it would not have been possible to find the right
scaling for the y-axis.

The csv files from the plot digitizer were then processed using an in-house
python script to build the dataset for ML model training. Firstly, the script
zeroed the velocity profiles such that they started from 0 and inverted the
coarctation velocity values to the positive y-axis. Secondly, any non 0 values
during diastole were set to 0 and the velocity profile was then interpolated
for 200 or 350 steps within the time interval, depending on the length of the
time interval and number of cycles. Lastly, the associated vessel name
(ascending aorta, innominate artery, left common carotid artery, left
subclavian artery, coarctation and descending aorta), case (pre or post
intervention), and heart rate were captured . A training dataset was then
generated from this information in the form of a table.

###  2.2 Machine Learning Pipeline

Our training dataset consisted of 3 numerical values; namely time in seconds,
velocity in metres per second and heart rate in beats per second.The Case and
Vessel features/ columns comprised of categorical or non-numeric entries.
Additional pre-processing steps were employed before ML model training.
Firstly, the categorical entries Case and Vessel were converted to numeric
entries using Scikit-learns label encoding function sklearn , as the models
within the Scikit-learns library require numerical inputs. For the numeric
entries (in particular the velocity values) it was found that the distribution
of the data was skewed to the right and contained several outliers. As was
stated in the previous paragraph the velocity values during diastole were set
to 0. Therefore, the presence of several 0 values contributed to skewing the
distribution and thus several outliers when a box and whisker plot was
plotted. Whilst the high velocity values at the tail end were due to the
coarctation. This would cause poor model performance for models that are
sensitive to outliers and lead to higher error on ML model predictions.
However, these outliers were biologically plausible and not as a result of
noise or errors, thus, a Log transform was applied to the velocity entries to
reduce the effect of the outliers using Numpy’s log1p function numpy , to
improve ML model performance but simultaneously preserve the outlier values
within training.

(a) Coarctation measurement post intervention

(b) Coarctation measurement pre intervention

(c) Innominate Artery measurement post intervention

(d) Ascending Aorta measurement pre intervention

Figure 2: Sample Doppler TTE images that were used within the study

The class of ML problem posed is a regression problem as the objective of the
ML model is to predict a continuous value given a set of discrete inputs.
Recalling that the main challenge faced was due to the discontinuity in the
time of measurement between vessels, the heart rate at the current measured
vessel varied from the previous vessel. As a result it was taken that the in-
vivo flow conditions at the measured vessel sites were not constant due to the
difference in heart rates. Which would then prove difficult when prescribing
BCs for CFD modelling as it is expected that the flow conditions in time are
the same at the prescribed vessels. It was noted earlier that changes in the
heart rate affect the velocity profile across the measured vessel, which
indicates a present relationship between heart rate and velocity. During CFD
modelling mass flow rate is the primary numerical BC parameter that is
prescribed at the inlets and outlets of the region of interest and is a
function of velocity. Therefore, the patients heart rate was seen as the key
input that could relate the in-vivo flow conditions in time with the velocity
which was the parameter needed to calculate the BCs. Thus, time (time interval
over which the velocity was measured), velocity (velocity across the measured
vessel site) and the heart rate (average heart across 3-4 cardiac cycles) were
the identified key numerical inputs. However, these values were noted to
change depending on which vessel was measured and case (whether an
intervention had occurred or not). To enable the ML models to take these
events into account the vessel name and case were included as categorical
variables. It is noted that although the vessel name which represents
anatomical position can be better represented by geometrical parameters which
could convey more insightful information, this in our case was not possible.
For geometric information was inconsistent within the given data, whilst the
vessel name was consistently given.

For this study the inputs to the ML models were time in seconds, the vessel
name (ascending aorta, innominate artery, left common carotid artery, left
subclavian artery, coarctation and descending aorta), case (pre or post
intervention) and heart rate in beats per second were used. Each of these
inputs were scaled using Scikit-learns Standard Scaler function sklearn . This
was to prevent error among models sensitive to scaling, such as linear
regression and support vector machines. The evaluation metric used to assess
model performance was the root mean square error (RMSE) hands-on . The
following models were trained within this study, namely; Linear Regression
model, which provided the baseline, Support Vector Machine Regressor, Random
Forest Regressor, Gradient Boosted Regressor, K-Neighbours Regressor (k-NN)
and a voting ensemble method. The train dataset was split into a train,
validation and test with 60%, 20%, and 20% split, respectively. Parameters for
the K-neighbours regressor, random forest regressor, gradient boosted
regressor, LightGBM regressor ke2017lightgbm  and support vector machine
regressor were optimized on the train set and validated against the validation
set sklearn . Finally, all models were then lastly fit to the combined
training and validation set, and evaluated on the test set.

###  2.3 CFD Pipeline

ANSYS Fluent (ANSYS Academic Research [Fluent], release 2020R2) was the
commercial software package used for the CFD study. A mesh constituting 2
million elements was generated usng the unrepaired geometry in
swanson2020patient . The number of elements used were greater than the minimum
number of elements required for grid independence for the used geometry, thus
the mesh was considered grid independent. The incompressible Navier-Stokes
equations were solved using finite volume method. While the pressure was
computed using PRESTO discretization scheme, the momentum was determined using
a 2nd order upwind scheme. Pressure and velocity were coupled using ANSYS
Fluent PISO scheme. For temporal discretization, a bounded 2nd order implicit
scheme was used with a time step of 0.01ms. Flow was modeled as laminar and
blood was assumed to be a Newtonian fluid with a density of 1060 kg/m3 and
viscosity of 0.004 kg/(m s). The vessel wall was modelled as rigid and a no-
slip boundary condition was applied. Convergence criteria of 10-3 was used
(ANSYS Academic Research [Fluent], release 2020R2, 37.21.1, Judging
Convergence, ANSYS, Inc.).

The maximum coarctation velocity output by ANSYS Fluent was compared to the
measured value from Doppler TTE, by calculating the percentage difference
between the two values. This was then used to validate the framework. The
reasoning is similar to that given in swanson2020patient , as Doppler TTE is
capable of reliably measuring the velocity of blood within the body and the
measurement was taken by an experienced sonographer. Although, pressure is the
clinically relevant flow parameter, the simplified Bernoulli method used by
the device has been shown to be inaccurate. Therefore, velocity was a more
reliable flow parameter for validating the framework.

To calculate the BCs, 5 ML models with the least RMSE were selected to
generate BCs for the CFD modelling process. A time interval of 0.442 seconds,
which corresponds to 135.6BPM, was used, as this was the calculated heart rate
at which the measurement across the coarctation was taken. The vessel names
and case (pre- intervention) were also input to each ML model. The CFD mass
flow rate input profiles were calculated from the velocity profiles output by
each ML model using, the stated density of blood and the vessel area. The set
of BCs were adapted from pirola2017choice ; madhavan2018effect ;
swanson2020patient  with the aim of assessing BCs that were commonly used in
literature whilst taking into account the limited data constraint due to the
use of Doppler TTE. Each set of BCs had to be simulated for the not adjusted
case (max velocity values obtained from Doppler TTE measurements with no
additional processing) and for the outputs of the selected 5 ML models. The
computational cost of running the simulations was therefore an important
factor.

Table 1 shows the set of BCs specified at the patient geometry inlet and
outlets for each ML model and the not adjusted case. The outlets consisted of
the arch branches (innominate artery; left common carotid artery, LCCA; left
subclavian artery, LSA) and descending aorta, abbreviated AoB and DAo
respectively. Note the outlets may also be refered to outlet 1-4 in the order
that they were given. Whilst the inlet comprising of the ascending aorta. 4
sets of BCs where used with the inlet BC having been kept constant and the
outlet BCs varied, as these BCs had greater impact on flow within the
coarctation region according to madhavan2018effect . BC 1, with zero pressure
at all outlets has been shown to result in the least accurate flow solution
when compared to other types of outlet BCs pirola2017choice . Thus, BC 1 was
primarily included as a benchmark case to provide the baseline performance of
the ML-CFD pipeline for each ML model and the not adjusted case. The zero
pressure BC with target mass flow adds a specified mass flow rate as a
constraint that must be met when calculating pressure at the outlet. This
provided an alternate way for pressure to be prescribed at the outlet, as mass
flow rate can be seen as an alternate form of prescribing velocity at the
outlets. As can be seen in BC 3 and BC 4.

Given that Madhavan et al showed that the effects of the inlet boundary
condition are only significant up to two inlet diameters distal to the inlet
patch madhavan2018effect , a plug flow velocity profile was assumed for the
inlet. Therefore, since the main region of interest, the coarctation, is
located further away this assumption was deemed adequate. However, the choice
of outlet BCs is crucial as they affect a wider range, of up to 5 diameters
distal to the outlet madhavan2018effect . Due to the constraint of limited
data imposed by the use of Doppler TTE, methods such as the 3 Element
Windkessel model (3-EWM) could not be applied. Instead, simpler outlet BCs
namely zero pressure and mass flow rate, were used. The ML model derived BCs
were compared with the not adjusted BCs, which were the BCs obtained from
Doppler TTE in swanson2020patient . Differences in the BCs used in the study
were largely due to the magnitudes of the values used.

BC | Inlet BC | AoB BC | DAo
---|---|---|---
BC 1 | mass flow rate | zero pressure | zero pressure
BC 2 | mass flow rate | mass flow rate | zero pressure
BC 3 | mass flow rate | zero pressure, target mass flow rate | zero pressure, target mass flow rate
BC 4 | mass flow rate | mass flow rate | zero pressure, target mass flow rate
|  |  |
Table 1: Boundary Conditions: This table contains the 4 sets of boundary
conditions (BCs) that were used within the study, namely BC 1-4. Outlets are
located at the arch branches (AoB) and the descending aorta, with the inlet at
the ascending aorta. The types of BCs used comprised of mass flow rate [kg/s],
zero pressure (static pressure [Pa] set to zero) and zero pressure target mass
flow rate (static pressure [Pa] set to 0 initially but constrained to meet a
specified mass flow [kg/s] value in subsequent iterations).

##  3 Results

This section begins with a brief description of the data processing results,
followed by the ML model results and finally the CFD results.

###  3.1 Data Pipeline and Clinical Data Results

The data processing pipeline output is a training dataset of size (3650 x 5).
The 5 columns are titled; time in seconds, velocity in meters per second, case
(pre or post intervention), vessel and the heart rate in beats per second.

###  3.2 ML Model Training Results

Table 2 shows the RMSE values for the ML models that were trained and
evaluated on the test set. Amongst the models, the k-NN regressor is found to
have fit the data best as it had the lowest RMSE. The voting regressor which
is an ensemble of the k-NN and random forest, performed second best. An
interesting observation is that the top 5 models primarily comprise tree-based
methods, with k-NN being the exception. However, it must be noted that the
RMSE is not the final metric with which the ML models are evaluated. Instead
the comparison of maximum velocity across the coarctation, obtained from ANSYS
Fluent, compared to the ground truth value obtained by Doppler TTE is also
used. The support vector machine regressor and linear regression had the
poorest results, as they had the highest RMSE values.

Model | RMSE[m/s]
---|---
Linear Regression | 0.4169
Support Vector Machine Regressor | 0.3695
Gradient Boosted Regressor | 0.2678
Light GBM Regressor | 0.1194
Random Forest Regressor | 0.0604
Random Forest Regressor Optimized | 0.0585
Voting Regressor | 0.0500
K-Nearest Neighbours Regressor | 0.0493
|
Table 2: ML Model Results: This table presents the root mean square error
(RSME) values for each machine learning model, when evaluated against the test
or hold out set. The test set constituted 20% of the original dataset built
for the study.

###  3.3 Boundary Condition Results

From Figure 3 we observe that we are able to obtain reasonable mass flow rate
values from each of the ML models. There is close agreement between the ML
model BCs at outlets 2 and 3, which on average deviate from the mean by 1% and
6%, respectively. Similarly, outlet 4 showed good agreement among the model
BCs with the exception being the LightGBM Regressor, which overestimated the
mass flow across the outlet. At outlet 1, all the ML model BCs underestimate
the flow in comparison to the not adjusted BC. It is at this outlet that we
observe the highest variation amongst the models at 19%. This is followed by
the inlet where the 2nd highest variation is observed at 10%. The LightGBM
Regressor outputs the highest mass flow rate, with the remaining ML models
outputting values less than that of the not-adjusted case.

Figure 3: Outlets 1 to 4 refer to the innominate artery, left common carotid
artery, left subclavian artery and descending aorta, respectively. We firstly
observe that the ML models output BCs whose magnitudes are physiologically
reasonable. At outlets 2 and 3 there is close agreement between the ML models
and the not adjust case, with percent deviations of 1% and 6%, respectively.
At outlets 1 and 4 less agreement is as higher percent deviations are observed
at 19% and 15%, respectively. Similarly, higher variation is observed at the
inlet but it is less than that at outlets 1 and 4. With regards to the ML
models, there are three notable trends. First, the LightGBM Regressor
overestimates the mass flow across the vessel except at outlet 1. Secondly,
the voting ensemble and Random Forest regressors (hyperparameter tuned and
default) output values with similar magnitudes. Whilst the k-NN regressor
across some vessels underestimates or over estimates values output by the
voting ensemble and Random Forest regressors.

###  3.4 Coarctation Velocity Results

(a) BC 1 results

(b) BC 2 results

(c) BC 3 results

(d) BC 4 results

Figure 4: Results for each type of boundary condition (BC) used.

Figure 4 presents the maximum coarctation velocities and percentage error (the
percentage difference between the ground truth max coarctation velocity 3.49
[m/s] and the output of the simulation) for each BC type and model used. At
first glance, it can be seen that the performance of the ML models varied
among BC 1-4, in Table 1. Of the ML derived BCs we observe that the k-NN model
BCs were found to have the least percentage error with the least being the
output of BC 2 with a value of 4.88%. Similarly the k-NN regressor using BC 4
output the 2nd most accurate result at 7%. The remaining ML models generally
performed poorly across all BCs, with the voting ensemble regressor with the
least percentage error outside that of the k-NN regressor. BC 1 (zero pressure
outlet BCs) resulted in the least accurate results except for the case of the
LightGBM regressor. As for this BC only the LightGBM showed marginal
improvement as compared to the not adjusted case. Between BC 2 to 3 the voting
, Random Forest (hyperparameter tuned and default) and LightGBM regressor’s
performance did not vary significantly. However, the k-NN regressor’s
performance varied significant across BC 1 and 3.

##  4 Discussion and Conclusion

Numerical modelling offers a means for improved analysis and understanding of
cardiovascular haemodynamics, through simulation of flow in complex
geometries. However, the inputs required for these methods can prove
challenging,particularly resource constrained settings. This study aimed to
add to the body of work in the intersection of both empirically derived
patient-specific BCs and applications of doppler echocardiography for patient
data acquisition pant2014methodological ; lassila2020population ;
swanson2020patient .

It is clear that the framework is capable of producing velocity results that
are physiologically accurate, whilst accounting for variations in the
patients’ heart rate. This is evidenced by the percentage difference within 5%
for BC2 and 7% for BC4 using the k-NN regressor. It is noteworthy that the
k-NN performed the best among the ML models. For apart from Linear Regression
it is the simplest amongst the ML models. In addition, this may suggest there
is a strong relationship between the input variables and the output based on
euclidean distance. Where as the poor performance of the tree-based methods
(voting ensemble, Random Forest, and LightGBM regressor’s) suggests that
underlying relationship that maps the input variables to the output may not be
achieved by asking a series of binary questions. As this is the underlying
assumption that tree-based models use. The poor performance of the tree based
methods also highlights that a low RMSE value may not result in an accurate
flow solution. Therefore, a more suitable metric for ML model selection prior
to calculating flow using CFD will be required. However, it at the same time
justifies the validation approach taken within the study, in that the solution
output by the CFD model was the final measure by which the framework was
validated.

The results suggest that mass flow rate outlet BCs are a feasible alternative
to pressure based 3-EWM which are traditionally favored within literature
pant2014methodological ; morbiducci2013inflow ; obaid2019computer , in cases
where pressure data is absent. For though 3-EWM has proven capable of
prescribing physiologically accurate BCs that result in highly accurate flow
solutions, it requires accurate patient pressure data and calibration of model
parameters such as resistance and capacitance. Although there are approaches
such as the one presented in pant2014methodological , which are capable of
calibrating 3-EWM parameters in cases where limited data are available,
pressure data are still required. Thus, such an approach would have proved
difficult to implement in this study. It would be possible to obtain
parameters from literature with similar studies, however, this would still
introduce errors due to variability between patients. Therefore, this approach
opens the door for haemodynamic modelling not only in cases where Doppler TTE
is used as the velocity data acquisition modality but also those where
pressure data may not be available.

###  4.1 Limitations

The dataset built for this study was limited as it was restricted to one
patient and the 4 additional Doppler TTE images that were provided. Therefore,
the generalizability of the ML models is limited. Extrapolating for cases
beyond those used within the study, may result in a performance drop of the
ML-CFD pipeline. Furthermore, the study was limited in the type of cases
modelled. Modelling for intra-patient variability, for example when the
patient is in a stressed state, was not possible. Therefore, modelling was
limited to a range of heart rates that capture a relaxed state. Both
lassila2020population  and pant2014methodological  note that there is a
significant difference between flow parameters for the two different states,
with pant2014methodological  further noting increased difficulty in accurately
modelling flow. Similarly, the effects of different patient patient geometries
on the performance of the framework cannot be evaluated. Also noting that
among the inputs used to train the ML models, the relative importance between
vessel names was not taken into account when label encoding was used.
Therefore, useful correlations from the relative importance were taken into
account. Lastly, the maximum coarctation velocities results show that the
choice of ML model used (and the type of BC prescribed at the outlet) had an
impact on the outcomes. This then motivates the need to evaluate the
performance of other models such as artificial neural networks (ANNs) that may
be able to determine more complex relationships within the data
feiger2020accelerating ; yevtushenko2021deep ; raissi2019pinn .

Assumptions made in the CFD modelling also contributed to the limitations of
the study. These include, the use of fixed BCs and the rigid wall assumption.
However, accounting for these considerations comes at increased computational
cost, hence their inclusion should be balanced against improvements in
accuracy pant2014methodological . A suitable compromise to be investigated is
including these factors in the ML model component. In their paper,
rackauckas2020universal  motivate for the application and advantage of
universal partial differential equations, which is a hybrid approach that
combines physics based and ML models. Thus, future work would be the
development of a hybrid ML and physics based lumped parameter model that
accounts for the noted limitations.

Supplementary information Acknowledgments Computations were performed using
facilities provided by the University of Cape Town’s ICTS High Performance
Computing team: hpc.uct.ac.za

## Declarations

  * •

Conflict of interest: The authors declare that they have no conflict of
interest.

  * •

Ethics approval: The study used secondary data from the Red Cross War Memorial
Children’s Hospital. Patient consent was obtained.

  * •

Code availability: The code used within this study is available at the given
link: https://figshare.com/s/a86bf491ef0a5b1e38cb

  * •

Authors’ contributions: VMMP contributed to the design of the pipeline,
methodology, analysis and writing of the manuscript. MN, AKM contributed to
the conceptualization of the work, analysis, methodology and critical revision
of the manuscript. TA, JL and LZ contributed to data collection and critical
revision of the manuscript.

##  Appendix A Boundary Condition Calculations

This appendix contains additional details on how the boundary conditions were
calculated to enable easy reproducibility of the framework discussed within
this paper.

The boundary conditions were calculated by first identifying the time at which
peak flow at the inlet occurred. This time value was then used to obtain the
mass flow rate values that occurred at this time. The initial value at the
inlet was then replaced by the sum of the outlet mass flow rates in order to
conserve mass continuity. This was firstly because within the body due to the
compliance of the vessels mass is not conserved and the models were trained on
physiological data the conservation law would not be learned. Secondly, doing
this resulted in values which were less erroneous as conservation of mass is
an assumption made by the ANSYS Fluent pressure based solver (ANSYS Academic
Research [Fluent], release 2020R2). Figures below show the BC values used to
generate the CFD results in this study. The abbreviations mf, zp and zpt are
mass flow rate [kg/s], zero pressure and zero pressure with targeted mass flow
rate, respectively.

Figure 5: Not Adjusted BC Values Figure 6: Voting Ensemble BC Values Figure 7:
k-NN BC Values  Figure 8: Random Forest (Hyperparameter tuned) BC Values
Figure 9: Random Forest BC Values Figure 10: LightGBM BC Values

## References

  *   * (1) Liu, Y., Chen, S., Zühlke, L., Black, G.C., Choy, M.-k., Li, N., Keavney, B.D.: Global birth prevalence of congenital heart defects 1970–2017: updated systematic review and meta-analysis of 260 studies. International Journal of Epidemiology 48(2), 455–463 (2019) https://academic.oup.com/ije/article-pdf/48/2/455/28479664/dyz009.pdf. https://doi.org/10.1093/ije/dyz009
  * (2) Zühlke, L., Lawrenson, J., Comitis, G., De Decker, R., Brooks, A., Fourie, B., Swanson, L., Hugo-Hamman, C.: Congenital heart disease in low-and lower-middle–income countries: current status and new opportunities. Current cardiology reports 21(12), 1–13 (2019)
  * (3) Obaid, D.R., Smith, D., Gilbert, M., Ashraf, S., Chase, A.: Computer simulated “virtual tavr” to guide tavr in the presence of a previous starr-edwards mitral prosthesis. Journal of Cardiovascular Computed Tomography 13(1), 38–40 (2019)
  * (4) Cecchi, E., Giglioli, C., Valente, S., Lazzeri, C., Gensini, G.F., Abbate, R., Mannini, L.: Role of hemodynamic shear stress in cardiovascular disease. Atherosclerosis 214(2), 249–256 (2011). https://doi.org/10.1016/j.atherosclerosis.2010.09.008
  * (5) Swanson, L., Owen, B., Keshmiri, A., Deyranlou, A., Aldersley, T., Lawrenson, J., Human, P., De Decker, R., Fourie, B., Comitis, G., et al.: A patient-specific cfd pipeline using doppler echocardiography for application in coarctation of the aorta in a limited resource clinical context. Frontiers in Bioengineering and Biotechnology 8, 409 (2020)
  * (6) Capelli, C., Sauvage, E., Giusti, G., Bosi, G.M., Ntsinjana, H., Carminati, M., Derrick, G., Marek, J., Khambadkone, S., Taylor, A.M., et al.: Patient-specific simulations for planning treatment in congenital heart disease. Interface Focus 8(1), 20170021 (2018)
  * (7) Pirola, S., Jarral, O., O’Regan, D., Asimakopoulos, G., Anderson, J., Pepper, J., Athanasiou, T., Xu, X.: Computational study of aortic hemodynamics for patients with an abnormal aortic valve: The importance of secondary flow at the ascending aorta inlet. APL bioengineering 2(2), 026101 (2018)
  * (8) Antonuccio, M.N., Mariotti, A., Fanni, B.M., Capellini, K., Capelli, C., Sauvage, E., Celi, S.: Effects of uncertainty of outlet boundary conditions in a patient-specific case of aortic coarctation. Annals of biomedical engineering 49(12), 3494–3507 (2021)
  * (9) Brüning, J., Hellmeier, F., Yevtushenko, P., Kühne, T., Goubergrits, L.: Uncertainty quantification for non-invasive assessment of pressure drop across a coarctation of the aorta using cfd. Cardiovascular engineering and technology 9(4), 582–596 (2018)
  * (10) Campbell, I.C., Ries, J., Dhawan, S.S., Quyyumi, A.A., Taylor, W.R., Oshinski, J.N.: Effect of inlet velocity profiles on patient-specific computational fluid dynamics simulations of the carotid bifurcation. Journal of biomechanical engineering 134(5) (2012)
  * (11) Conti, M., Long, C., Marconi, M., Berchiolli, R., Bazilevs, Y., Reali, A.: Carotid artery hemodynamics before and after stenting: A patient specific cfd study. Computers & Fluids 141, 62–74 (2016)
  * (12) Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, ??? (2016). https://books.google.co.za/books?id=kOXDtAEACAAJ
  * (13) Feiger, B., Gounley, J., Adler, D., Leopold, J.A., Draeger, E.W., Chaudhury, R., Ryan, J., Pathangey, G., Winarta, K., Frakes, D., et al.: Accelerating massively parallel hemodynamic models of coarctation of the aorta using neural networks. Scientific reports 10(1), 1–13 (2020)
  * (14) Nita, C.-I., Puiu, A., Bunescu, D., Mihai Itu, L., Mihalef, V., Chintalapani, G., Armstrong, A., Zampi, J., Benson, L., Sharma, P., et al.: Personalized pre-and post-operative hemodynamic assessment of aortic coarctation from 3d rotational angiography. Cardiovascular Engineering and Technology 13(1), 14–40 (2022)
  * (15) Yevtushenko, P., Goubergrits, L., Gundelwein, L., Setio, A., Heimann, T., Ramm, H., Lamecker, H., Kuehne, T., Meyer, A., Schafstedde, M.: Deep learning based centerline-aggregated aortic hemodynamics: An efficient alternative to numerical modelling of hemodynamics. IEEE Journal of Biomedical and Health Informatics (2021)
  * (16) Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics 378, 686–707 (2019). https://doi.org/10.1016/j.jcp.2018.10.045
  * (17) Madhavan, S., Kemmerling, E.M.C.: The effect of inlet and outlet boundary conditions in image-based cfd modeling of aortic flow. Biomedical engineering online 17(1), 1–20 (2018)
  * (18) Morbiducci, U., Ponzini, R., Gallo, D., Bignardi, C., Rizzo, G.: Inflow boundary conditions for image-based computational hemodynamics: impact of idealized versus measured velocity profiles in the human aorta. Journal of biomechanics 46(1), 102–109 (2013)
  * (19) Pirola, S., Cheng, Z., Jarral, O., O’Regan, D., Pepper, J., Athanasiou, T., Xu, X.: On the choice of outlet boundary conditions for patient-specific analysis of aortic flow using computational fluid dynamics. Journal of biomechanics 60, 15–21 (2017)
  * (20) Gèron, A.: Hands-on Machine Learning with Scikit-Learn and TensorFlow : Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly Media, Sebastopol, CA (2017)
  * (21) PlotDigitizer: Version 2.2 (2022). https://plotdigitizer.com
  * (22) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in python. Journal of machine learning research 12(Oct), 2825–2830 (2011)
  * (23) Harris, C.R., Millman, K.J., van der Walt, S.J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N.J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M.H., Brett, M., Haldane, A., Fernández del Río, J., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., Oliphant, T.E.: Array programming with NumPy. Nature 585, 357–362 (2020). https://doi.org/10.1038/s41586-020-2649-2
  * (24) Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.-Y.: Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 30 (2017)
  * (25) Pant, S., Fabrèges, B., Gerbeau, J.-F., Vignon-Clementel, I.: A methodological paradigm for patient-specific multi-scale cfd simulations: from clinical measurements to parameter estimates for individual analysis. International journal for numerical methods in biomedical engineering 30(12), 1614–1648 (2014)
  * (26) Lassila, T., Sarrami-Foroushani, A., Hejazi, S., Frangi, A.F.: Population-specific modelling of between/within-subject flow variability in the carotid arteries of the elderly. International Journal for Numerical Methods in Biomedical Engineering 36(1), 3271 (2020)
  * (27) Rackauckas, C., Ma, Y., Martensen, J., Warner, C., Zubov, K., Supekar, R., Skinner, D., Ramadhan, A., Edelman, A.: Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385 (2020)