# Dusk Till Dawn: Self-supervised Nighttime Stereo Depth Estimation using
Visual Foundation Models

Madhu Vankadari, Samuel Hodgson, Sangyun Shin∗, Kaichen Zhou∗, Andrew Markham,
and Niki Trigoni $*$ refers to equal contributionAll authors are with the
University of Oxford, Oxford, UK

###### Abstract

Self-supervised depth estimation algorithms rely heavily on frame-warping
relationships, exhibiting substantial performance degradation when applied in
challenging circumstances, such as low-visibility and nighttime scenarios with
varying illumination conditions. Addressing this challenge, we introduce an
algorithm designed to achieve accurate self-supervised stereo depth estimation
focusing on nighttime conditions. Specifically, we use pretrained visual
foundation models to extract generalised features across challenging scenes
and present an efficient method for matching and integrating these features
from stereo frames. Moreover, to prevent pixels violating photometric
consistency assumption from negatively affecting the depth predictions, we
propose a novel masking approach designed to filter out such pixels. Lastly,
addressing weaknesses in the evaluation of current depth estimation
algorithms, we present novel evaluation metrics. Our experiments, conducted on
challenging datasets including Oxford RobotCar and Multi-Spectral Stereo,
demonstrate the robust improvements realized by our approach. Code is
available at https://github.com/madhubabuv/dtd

##  I Introduction

Depth estimation is a pivotal subject within computer vision, with wide-
ranging implications for applications such as autonomous driving, augmented
and virtual reality, and robotics [1, 2]. Despite the accomplishments of
supervised depth estimation algorithms, these methods typically depend on
high-resolution ground truth data - a challenge that requires substantial
computational resources, costly 3D LiDAR sensors, and heavy computational
requirements [3, 4].

Addressing the need for ground-truth data, recent research has shown interest
in self-supervised depth estimation methods [5, 6]. Such approaches typically
warp the source frame to the target frame using the learned depth information.
A photometric loss is found between the reconstructed and actual target images
to constrain the learning process. While monocular depth estimation algorithms
are widely applicable, they often lack scale information [7] and exhibit
limited generalizability [8]. In contrast, self-supervised stereo depth
estimation algorithms that use the correspondence between left and right
frames yield more robust performance [9].

Figure 1: A comparison of the estimated disparity using our method (Ours) with
a SOTA stereo-matching method, IGEV-Stereo [10]. Note how the sky (blue
rectangle) is incorrectly estimated by IGEV-Stereo as being very near.
Similarly, there is a lack of detail showing the edge of the wall and the
lamp-post (green rectangle). In comparison, our method is able to accurately
estimate these depths.

Self-supervised stereo depth estimation, however, relies on photometric
consistency assumptions and conventional warping relationships, which are
constrained by favorable lighting conditions. These assumptions break down in
nighttime scenarios, characterized by low texture and fluctuating illumination
[11]. Traditional approaches primarily depend on pretrained classification
networks, such as [12], for feature extraction. The utility of these networks
is confined to relatively small datasets due to the need for labeled data, and
their results are domain-specific. Contemporary self-supervised feature
extractors offer enhanced robustness and clarity in feature mapping, as they
do not rely on labeled data [13, 14]. Conventional stereo depth estimation
techniques also tend to combine stereo features indiscriminately, without
addressing regions that have correspondence issues [9, 15].

In light of this, our approach concentrates on obtaining accurate self-
supervised stereo depth estimation at night. The contributions of our paper
can be summarized as follows:

  * •

We present an architecture capable of efficient self-supervised stereo depth
estimation at night, using visual foundation models and a photometric loss
function.

  * •

We introduce a feature-level mask to mitigate the impact of pixels violating
illumination assumptions.

  * •

We propose a distance regularizer aimed at enhancing the feature descriptions
to estimate accurate depth maps.

  * •

We provide a more rational set of evaluation metrics to assess the performance
of depth estimation.

##  II Related Work

Photometric consistency assumes the scene to be static, free of any noise or
occlusions, lambertian and temporally illumination invariant. Various changes
in the appearance of scenes during nighttime cause systems trained only for
daytime datasets to fail. Approaches are required to deal with the scene
lighting issues, such as the lack of the sun as a primary source of light.

###  II-A Nighttime Self-supervised Depth Estimation

Loosely, we classify previous work as being based either on domain adaptation
or image enhancement. Adaptation-based approaches seek to overcome the domain
shift between day and night using synthesised data to align day/night or
clear/inclement weather image encodings [16, 11]. They aim to obtain the same
features across different conditions. Synthesised data is created using
standard datasets (i.e. [17]), and a Generative Adversarial Network (GAN).
[18] uses a Monodepth2 [4] architecture with an adversarially-trained
nighttime image encoder. Features are aligned using a GAN to generate day
images from night, and a discriminator to enforce similarity. [19] emulates
this approach, but trains in the output space as well as the feature space.
[20] uses the GAN outputs to train two feature extractors, a day-night
invariant feature extractor that forms the backbone for depth prediction, as
well as night and day style extractors for training. Outside of our broader
categorisation, [21] uses a feature space (rather than image space)
contrastive loss to improve domain generalisation beyond that of [4].
Enhancement-based approaches aim to improve performance over day models by
taking greater account of scene lighting, for example, by isolating
illumination information within the image. [22] uses a learned image
enhancement and adapted masking from [4]. [23] estimates the illumination
change between the consecutive nighttime images to relax assumptions of
photometric loss for better depth estimation.

###  II-B All-day Self-supervised Depth Estimation

Concurrent state-of-the-art approaches [16] and [11] also consider weather
conditions in their approaches. Both adapt [4] for their all-day unified
networks, using GANs to augment for both weather and lighting conditions. [16]
uses a semi-augmented warping to minimise GAN-induced inconsistency between
consecutive frames, and uses raw inputs for pose estimation to minimise error
propagation. Similarly, [11] uses daytime depth estimation to distill depth
knowledge to nighttime using image translation networks. [24] uses a complex,
partially adversarial architecture, with a learned image enhancement to
estimate illumination and uncertainty, which is then masked from the loss.

###  II-C Supervised Stereo Depth Estimation

All approaches mentioned so far are monocular, even if stereo images are used
during training. In terms of supervised stereo depth estimation, [25] uses a
transformer and 4D correlation feature matrix, including an iterative
refinement inspired by [26], to derive depth, flow, and disparity. [27] uses a
coarse-to-fine approach with a hierarchical network and adaptive group
correlation for getting fine disparities. [28] builds an attention-based cost
volume to suppress redundant information and enhance matching-related
information. [10] unifies stereo and optical flow approaches based on 2D
convolution, avoiding the memory cost of 3D convolution. [29] trains a model
using day datasets that have ground truth depth and further uses domain
adaptation to work on day to night. Our method does not use any daytime data,
domain adaptation, or ground truth depth for training.

Figure 2: Our approach consists of four main elements. Features are encoded
independently for each input using DINO [13], a learnable projection head
adapts these features and reduces their dimension, giving $f_{l}$ and $f_{r}$.
Stereo matching of the features then takes place, with disparity filtering
yielding the mask $\mathcal{M}$, and the combination of $f_{l}$ and $f_{r}$
providing the correspondence volume $\mathcal{C}_{disp}$. $\mathcal{W}_{disp}$
is found by using softmax on $\mathcal{C}_{disp}$, which is used to find
coarse disparity $d_{c}$. Coarse disparity and the mask combine to give global
disparity $d_{g}$, which is refined and upsampled to give final depth.

##  III Proposed Method

Given a pair of stereo images $(I_{l},I_{r})$, we aim to estimate the per-
pixel depth map $\mathbf{d}$ using self-supervised learning. The proposed
framework is depicted in Fig 2. Our method is composed of 4 main components,
namely the feature extractor, projection head, stereo matcher, and upsampler.

###  III-A Feature Extractor

In contrast to existing approaches, we use visual foundation models, DINO [13]
and DINOv2 [14], as feature extractors for the input images. The models are
trained on the ImageNet [30] and Facebook LVD-142M [14] datasets,
respectively. In our experiments, we use the pretrained _small_ -models of
DINO and DINOv2, with patch sizes of $8$ and $14$, respectively. For the rest
of the paper, we refer to DINO ViT-S/8 (patch size 8) as the encoder unless
otherwise stated. Our encoder has a _conv_ -layer followed by a series of 12
_transformer_ layers. Given an image $I\in\mathbb{R}^{H\times W\times 3}$ with
height $H$ and width $W$, the conv-layer converts the image into overlapping
$8\times 8$ patches with stride $4$, resulting in a feature-map
$\hat{f}\in\mathbb{R}^{h^{\prime}\times w^{\prime}\times 384}$ where
$h^{\prime}=H/4$, $w^{\prime}=W/4$. After processing $\hat{f}$ through the
first 6 transformer layers, the estimated output feature-map
$\tilde{f}\in\mathbb{R}^{h^{\prime}\times w^{\prime}\times 384}$ is subsampled
with stride 2 in the height and width dimensions, giving an $8\times$
downscaled feature map that is further processed by 6 more transformer layers
resulting $f\in\mathbb{R}^{h\times w\times 384}$, where $h=H/8$, $w=W/8$.

Selection of the transformer layers is based on [31]. They observe that
various encoder layers act similarly to hierarchical CNN layers in capturing
local-to-global information as the depth of the network increases. Deeper
layers capture semantics, and shallow layers capture local details (including
positions). Middle layers tend to carry both. For accurate disparity
estimation, we use both deep and middle layer features, $(f,\tilde{f})$, for
stereo matching.

###  III-B Projection Head

The projection head takes feature vectors $f,\tilde{f}$ from the encoder and
projects them into lower dimensional space $\mathbb{R}^{D}$ where $D<384$.
This is done because (1) when PCA is performed on the $384$ dimensions, we
observe the 10 first principal components to explain more than $50\%$ of the
total variance, suggesting it is safe to project to lower dimensional space
and (2) the computational complexity of the stereo matcher during inference
and training will be reduced.

The projection head consists of two _conv_ -layers with kernel size as
$1\times 1$, and $\mathrm{ReLU}$ as activation function in the middle. We use
$D=128$ as the output description dimension and the same projection head for
both left and right features.

###  III-C Stereo Matcher

The stereo matcher takes the multi-scale left and right features
$\\{(f_{l},f_{r}),(\tilde{f}_{l},\tilde{f}_{r})\\}$ as input and estimates the
disparity map $\tilde{d}_{r}\in\mathbb{R}^{h^{\prime}\times w^{\prime}\times
1}$ in three stages. First, features are enhanced with cross-image feature
context using a transformer module. Then, a disparity map $d_{g}$ is estimated
at $1/8$ scale using the $(f_{l},f_{r})$ features and global matching. Lastly,
the disparity is bilinearly interpolated by $2\times$ and corrected for the
interpolation artifacts with a residual disparity estimated using
$(\tilde{f}_{l},\tilde{f}_{r})$ and local matching. These modules are
explained in detail in the following sections.

Transformer: Stereo images are processed through the encoder and projection
head individually, meaning no cross-image interaction or information exchange.
In previous works [25, 32, 33], it is suggested that adding cross-image
information results in better matching accuracy. We therefore use a
transformer module similar to [25] with fixed positional encoding after the
projection head. To further reduce computational complexity, we only aggregate
features that are on the respective epipolar line for any given feature. We
use rectified stereo-images that make the search problem 1D, as the epipolar
lines are strictly straight. Given $f_{l}$ and $f_{r}$, the transformer module
outputs $\textbf{f}_{l}$ and $\textbf{f}_{r}$ in $R^{h\times w\times D}$.

Coarse Disparity Estimation: The output features from the transformer module
are used to compute the dense correspondence volume $C_{disp}$ using
normalized feature correlation (i.e, global matching) with a simple matrix
multiplication

$\mathcal{C}_{disp}=\frac{\textbf{f}_{l}\cdot\textbf{f}_{r}^{\top}}{\sqrt{D}}\in\mathbb{R}^{h\times
w\times w}.$ (1)

We obtain the matching distribution $\mathcal{W}_{disp}$ using a softmax over
the last dimension of $\mathcal{C}_{disp}$. This is then multiplied with a 1D
pixel grid $\mathcal{P}_{\text{1D}}\in\mathbb{R}^{w}$ to obtain the
corresponding pixel locations $\mathcal{G}_{\text{1D}}\in\mathbb{R}^{h\times
w}$. Finally, the coarse disparity $d_{c}$ can be computed as the difference
between $\mathcal{P}_{\text{1D}}$ and $\mathcal{G}_{\text{1D}}$. Formally,
this can be written as:

$\begin{gathered}\mathcal{W}_{disp}=\mathrm{softmax}(C_{disp}),\,\text{and}\,\,\,\mathcal{G}_{\text{1D}}=\mathcal{W}_{disp}\mathcal{P}_{\text{1D}},\\\
d_{c}=\mathrm{ReLU}(\mathcal{G}_{\text{1D}}-\mathcal{P}_{\text{1D}}),\end{gathered}$
(2)

where $\mathrm{ReLU}$ ensures the disparity is always positive.

Figure 3: The qualitative comparison of the proposed method with SGM [34] and
the state-of-the-art supervised methods Unimatch-Stereo [25], IGEV-Stereo
[10], and Sharma et al. [29]. The brighter the pixel is, the closer it is to
the camera.

Disparity Filtering: One observation made during our experiments is that
features belonging to areas of low texture, particularly the sky, resulted in
very noisy disparity estimates similar to [34, 10] as shown in Fig. 1. We
explain this as being due to reduced camera sensitivity in low-light causing
an accumulation of noise, resulting in erroneous and noisy features. Such
features exacerbate incorrect feature matches, causing noisy estimates, as we
are extracting disparity as a byproduct, rather than estimating it directly as
in [29]. To address this, we propose a simple yet effective solution based on
intra-image feature description distances, using them to mask the noisy areas.
We conjecture that features belonging to noisy areas tend to have a lower
minimum distance from their nearest neighbors, compared to the features that
belong to well-lit areas. In a set of $n$ features, given a feature $f_{i}\in
f$ with $i\in\\{1,2,3...,n\\}$: (1) we normalize to have unit length; (2) we
estimate one to all cosine similarity to find the nearest neighbor $f_{j}$,
with $j\neq i$; and (3) we calculate $l_{2}$ distance $p_{i}$ between $f_{i}$
and $f_{j}$. Formally, this is,
$f^{\prime}_{j}=\mathrm{argmax}_{j}(f^{\prime}_{i}\cdot
f^{\prime\top}_{j}),\,j\neq i$ where $f^{\prime}=\frac{f}{\lvert\lvert
f\lvert\rvert}$ and
$p_{i}=\left|\left|f^{\prime}_{i}-f^{\prime}_{j}\right|\right|$. Estimated
distances are then used to filter good disparity values from noisy ones by
estimating a mask $\mathcal{M}$, which is used to find masked disparity
$d_{m}$ as:

$d_{m}=\mathcal{M}\cdot d_{c},\quad\mathcal{M}=\begin{cases}1,&\text{if
}p_{i}>\zeta,\\\ 0,&\text{otherwise},\end{cases}$ (3)

where $\zeta$ is set to $0.2$ in our experiments.

Disparity Propagation Some poorly lit regions that contain useful information
may get lost during masking. This creates holes in the estimated disparity.
We, therefore, propagate the disparity at the valid pixels to the masked-out
pixels by measuring self-similarity, as in [25]. This is done using an
attention layer to find global disparity, $d_{g}$:

$d_{g}=\mathrm{softmax}\left(\frac{f_{l}\cdot
f_{l}^{\top}}{\sqrt{D}}\right)\cdot d_{m}\,$ (4)

Disparity Refinement: The current global disparity $d_{g}$ is at $1/8\times$
resolution. To increase the resolution, we upsample $d_{g}$ using bilinear
interpolation by $2\times$. Doing so inevitably introduces interpolation
artifacts, so we use fine-level features $\tilde{f}_{l},\tilde{f}_{r}$ to
account for this. We first warp the right-features $\tilde{f}_{r}$ onto the
left frame using the upsampled disparity $d_{g}^{up}$. The original left-
features $\tilde{f}_{l}$ and the warped-features $\tilde{f}^{\prime}_{l}$ are
then passed through the transformer, disparity estimation, and filtering to
estimate residual disparity $\nabla d$. Using localised attention on
subsampled features in the refinement process is suggested in [25] to improve
accuracy through local feature interactions. The residual disparity is added
to the upsampled disparity, giving $d_{r}=d^{up}_{g}+\nabla d$. Finally, the
refined disparity $d_{r}$ is correlated with $\tilde{f}_{l}$ to output
disparity $\tilde{d}_{r}\in\mathbb{R}^{h^{\prime}\times w^{\prime}\times 1}$.

###  III-D Upsampling

The disparity output $\tilde{d}_{r}$ is upsampled to give final output
disparity $d\in\mathbb{R}^{H\times W\times 1}$. Instead of bilinear
interpolation, which can result in blurry borders, we use the learnable convex
upsampling method proposed by RAFT [26]. In it, two (3 $\times$ 3)
convolutional layers are used to predict a mask and perform softmax over the 9
neighbours of a given pixel.

Metric | Method | type |  Abs. Rel. $(\downarrow)$ |  Sq. Rel. $(\downarrow)$ |  RMSE $(\downarrow)$ |  Log RMSE $(\downarrow)$ | $\delta<1.25(\uparrow)$ | $\delta<1.25^{2}(\uparrow)$ | $\delta<1.25^{3}(\uparrow)$
---|---|---|---|---|---|---|---|---|---
U |  SGM [34] | - | 0.237 | 3.453 | 8.393 | 0.358 | 0.689 | 0.862 | 0.924
UniMatch-Stereo [25] | Sup | 0.207 | 2.521 | 9.087 | 0.373 | 0.588 | 0.793 | 0.906
IGEV-Stereo [10] | Sup | 0.147 | 1.655 | 7.092 | 0.312 | 0.782 | 0.888 | 0.934
|  Sharma et al. [29] | Sup | 0.225 | 1.728 | 6.489 | 0.278 | 0.669 | 0.920 | 0.963
| Ours | Self-Sup | 0.177 | 1.970 | 7.077 | 0.274 | 0.744 | 0.900 | 0.951
W |  SGM [34] | - | 0.246 | 3.711 | 9.313 | 0.374 | 0.630 | 0.825 | 0.900
Unimatch-Stereo [25] | Sup | 0.278 | 4.379 | 10.237 | 0.426 | 0.422 | 0.660 | 0.828
IGEV-Stereo [10] | Sup | 0.184 | 2.649 | 7.433 | 0.327 | 0.703 | 0.830 | 0.894
|  Sharma et al. [29] | Sup | 0.229 | 2.113 | 6.750 | 0.284 | 0.639 | 0.892 | 0.945
| Ours | Self-Sup | 0.192 | 2.427 | 7.100 | 0.275 | 0.703 | 0.870 | 0.931
TABLE I: Quantitative evaluation of our proposed method against the SOTA. This
evaluation is carried out with a maximum depth range of 50 meters. U refers to
the unweighted metric, and W to the proposed weighted metric. In the type-
column, “sup” refers to supervised training, and “self-sup” refers to self-
supervised training. Bold shows the best performance and underline refers to
second best results.

###  III-E Training Losses

Photometric Loss: Estimated disparity is used to reconstruct the left image
from the right using bilinear interpolation. The reconstructed image is
compared with the original left image to calculate the photometric loss. Given
the left and right-images $(I_{l},I_{r})$ and estimated disparity $d$, the
photometric loss $L_{\rm photo}$ is:

$\displaystyle\mathbf{d}=\frac{b\lambda}{d},\,\,\,\hat{I}_{l}=\mathrm{warp}(I_{r},\mathbf{d},K,T_{lr}),$
(5) $\displaystyle L_{\rm photo}=\alpha\lvert
I_{l}-\hat{I}_{l}\rvert+(1-\alpha)\mathrm{SSIM}(I_{l},\hat{I}_{l}),$ (6)

where $\mathbf{d}$ is the output depth map, $b$ is the baseline distance
between the left and right cameras, $\lambda$ is the focal length, $K$ is the
camera calibration matrix, $T_{lr}\in SE(3)$ is extrinsics of the stereo-rig,
$\alpha$ is the convex combination weight between $L_{1}$ and $SSIM$ losses,
and is set to $0.15$. The $\mathrm{warp}(\cdot)$ function warps the left from
the right image using $\mathbf{d},K$ and $T_{lr}$. More details of this loss
can be found in [4].

Distance Regularizer: We also encourage all of the features to maximise the
minimum distance from their nearest neighbour using a regularization loss
inspired by [35]. This allows the features from poorly lit areas to improve.
However, there is an imbalance between well and poorly-lit areas in the
majority of our training split images, similar to the class imbalance problem
from classification literature. This is addressed by using a modulation
factor, $\gamma$, to reduce the concentration of loss on features that already
have higher minimum distance with their nearest neighbour, and to focus more
on small feature distances. Formally:

$L_{\rm reg}=-\frac{1}{n}\sum_{i=0}^{n}(1-p_{i})^{\gamma}\log(p_{i}),$ (7)

where $\gamma$ is a modulation factor similar to focal-loss in [36], used here
to focus more on features that have low minimum distances. We set $\gamma=2$
in our experiments.

In order to make the estimated disparity spatially smooth while preserving the
edges, the common edge-aware Disparity Smoothness Loss $L_{\rm smooth}$ from
[3] is used. Finally, the total loss is $L_{\rm total}=L_{\rm
photo}+\beta_{1}L_{\rm reg}+\beta_{2}L_{\rm smooth}$, where $\beta_{1}$
balances how much we spread the features on the unit-sphere and is set as
$\beta_{1}=1$, and we choose $\beta_{2}=0.1$.

##  IV Experiments

###  IV-A Datasets

Throughout our experiments, we train on the RobotCar Dataset [17] and test on
both the Robotcar [17] and MS2 [37] datasets. Details of each are given below.

Oxford RobotCar: The Oxford RobotCar Dataset [17] is an autonomous driving
dataset collected on the same route over a year in Oxford, UK. We follow the
data splits proposed in [23] to exclude the geographical overlaps between
training and test splits. We use the six sequences from the traverse on
2014-12-16-18-44-24 for our experiments, providing $19612$ images for
training, $4559$ images for validation, and $709$ images for testing. Ground
truth depth data for evaluation is generated by projecting the LiDAR data from
several nearby frames into the test frame using the available RTK
111Quantitative results may change when other forms of pose data such as VO or
INS is used to generate ground truth depth data.

Multi-Spectral Stereo (MS2) Dataset: The MS2 dataset [37] contains 184K pairs
taken from multi-spectral sensors on a vehicle in Daejeon, South Korea. The
sequences include various lighting, weather, and traffic conditions. Following
the evaluation split proposed in [37], we use the Road, City, and Campus
nighttime sequences, further sub-sampling them with $5\rm m$ distance between
consecutive test images. This gave 1,470 pairs for evaluation. We use the
(filtered) ground truth depth data released with the dataset for the
evaluation.

###  IV-B Training details

The framework is trained using the Robotcar dataset for 20 epochs with an
input image resolution of $192\times 320$. We used a batch size of $8$ and the
Adam [38] optimizer with a constant learning rate of $1e-4$.

Figure 4: The visualization of (a) the ground truth depth distribution of the
Robotcar test split, and (b) square relative error calculated at different
depth-bins using the proposed weighted metric.

###  IV-C Baselines:

To the best of our knowledge, there is no self-supervised system that
estimates depth from night-time stereo-images. We therefore compare our method
with a classical method, Semi-Global Matching (SGM) [34], and 3 state-of-the-
art supervised methods: UniMatch-Stereo222We used their in-the-wild use
stereo-matching with refinement model from GitHub during the evaluation [25],
IGEV-Stereo [10], and Sharma et al. [29]. Note that these methods are trained
end-to-end only for the purpose of stereo-matching, with large amounts of
ground truth data. Also, we found that the disparity estimation of Unimatch-
Stereo and IGEV-Stereo drops drastically when tested at the same resolution as
ours. Therefore, we use $2\times$ more resolution while reporting their
results.

###  IV-D Depth Evaluation

The depth-estimation performance is evaluated qualitatively and quantitatively
on both the Robotcar [17] and MS2 [37] datasets. Quantitative evaluation is
carried out using the metrics proposed in [39]. For every test image, existing
methods compute their metrics as the mean of all valid pixels up to a given
depth range ($50\rm m$ in our evaluation), usually only with a sparse set of
LiDAR points. Taking the overall mean is sensible when pixels are uniformly
distributed over the depth range, however, this is not the case for the
RobotCar dataset [17] as shown in Fig 4(a). The same effect is observed for
most autonomous driving datasets, including the MS2 dataset [37]. This is due
to the fact that the majority of pixels in an image are occupied by points
that are very close to the camera. Evaluating performance on this kind of data
skews the evaluation, as the performance on nearby points outweighs the
performance on the far-away points. To mitigate this, we propose the use of
depth bins splitting the given depth range into $M$ bins. Letting $\mathrm{x}$
be a metric, unweighted ($U$) and weighted ($W$) evaluation metrics for a
given test image can be written as:

$\displaystyle\mathrm{x}_{\rm
unweighted}=\frac{1}{Z}\sum_{i=1}^{Z}\mathrm{x}_{i},$
$\displaystyle{\large\mathrm{x}_{\rm
weighted}=\frac{1}{M}\sum\limits_{N=1}^{M}\frac{1}{N_{i}}\sum\limits_{j=1}^{N_{i}}\mathrm{x}_{ij},}$
(8)

where $Z$ is the total number of valid pixels, $M$ is the total number of bins
and $N_{i}$ is the total number of valid pixels in the $i$-th depth-bin. We
set $M=10$, i.e., each bin covers $5\rm m$ depth. We reported the numbers from
both metrics, and we did not use eigen or garg crops [4] during our
evaluation.

Figure 5: Visualization of the estimated masks in (c), with their input-images
(Left camera) in (a), and the estimated disparity-maps in (b).

RobotCar: We give a qualitative comparison depicted in Fig 3. Our method is
able to extract crisp details, even in poorly illuminated regions. The effect
of the masking is clear when looking at the disparity estimated for the sky
pixels. We also visualize the estimated masks and the respective filtered
disparity maps in Fig. 5. Plausible masks are generated even for noisy low-
illumination areas such as the sky. The original ground truth depth images are
very sparse, making comparison hard. Hence, we dilated them for visualization.
Quantitative results are given in Table I. Our method performs on par with the
baselines in the majority of metrics across both variations despite being
self-supervised. Fig 4(b) shows the mean squared relative error of IGEV-Stereo
[10] and Ours for the test set. Per the unweighted metrics in Table I, IGEV-
Stereo [10] performs better. One can see, however, the performance clearly
degrades as the depth range increases compared to ours. This effect is much
better captured in the weighted metric. Similar observations are made for
other metrics as well. Lastly, note that our outstanding performance for
challenging regions, such as the sky, is not taken into account during the
quantitative evaluation due to the absence of the ground truth. As one can see
in Fig. 3, all other methods estimate valid depth (brighter pixels) for the
sky where they are supposed to be darker, as seen in the ground truth.

MS2: To further evaluate the generalisability of our method, we evaluated the
model trained on RobotCar dataset using the test split of the MS2 dataset with
the same $50\rm m$ maximum depth range. Despite differences in geographic
locations and lighting, and being trained on a relatively small dataset, our
model still estimates very plausible disparity maps and pixel masks as shown
in Fig 6. Quantitatively, our method performs better than SGM [34] and is
comparable to other methods as shown in Table II.

Metric | Method | Abs. Rel. | RMSE |  $\delta<$ 1.25 |  $\delta<$ $1.25^{3}$
---|---|---|---|---|---
U |  SGM [34] | 0.185 | 6.106 | 0.773 | 0.968
Unimatch-Stereo [25] | 0.095 | 3.396 | 0.910 | 0.994
IGEV-Stereo [10] | 0.099 | 3.918 | 0.894 | 0.985
|  Sharma et al. [29] | 0.193 | 5.077 | 0.713 | 0.990
| Ours | 0.182 | 5.838 | 0.740 | 0.977
W |  SGM [34] | 0.183 | 6.482 | 0.764 | 0.969
Unimatch-Stereo [25] | 0.112 | 4.182 | 0.860 | 0.991
IGEV-Stereo [10] | 0.125 | 4.977 | 0.815 | 0.976
|  Sharma et al. [29] | 0.180 | 5.489 | 0.733 | 0.987
| Ours | 0.180 | 6.162 | 0.716 | 0.978
TABLE II: Quantitative evaluation of our proposed method (trained on RobotCar) against the SOTA on MS2 Dataset Method |  Abs. Rel. $(\downarrow)$ |  RMSE $(\downarrow)$ | $\delta<1.25(\uparrow)$ | $\delta<1.25^{3}(\uparrow)$
---|---|---|---|---
Base model | 0.201 | 7.734 | 0.724 | 0.943
w/ Mask | 0.214 | 7.644 | 0.712 | 0.948
w/ Mask + Reg | 0.188 | 7.409 | 0.744 | 0.946
w/ DINO-V2 | 0.204 | 7.871 | 0.711 | 0.944
TABLE III: Ablation study showing the importance of different modules in our
system. This evaluation is carried out using the unweighted metrics with 50
meters as the maximum depth. Figure 6: Visualization of the estimated
disparities in (b) and the masks in (c) with their input-images (Left camera)
in (a) when tested on the unseen MS2 dataset.

###  IV-E Ablation Studies

We performed various ablation studies on RobotCar dataset to understand the
impact of different modules. The results are shown in Table III. Base model
uses DINO-V1 as the encoder, with stereo-matching and upsampling, trained
using $L_{\rm photo}$ alone. W/ mask had features masked before stereo-
matching, also trained with $L_{\rm photo}$ alone. W/ mask + reg used the
regularization loss $L_{\rm reg}$ with $L_{\rm photo}$. The improvement in
both error and accuracy metrics explains the importance of the proposed
masking and regularization loss for accurate depth estimation.

###  IV-F Failure Cases:

Interestingly, training with a DINOv2 [14] encoder yielded a performance drop
despite being pretrained on a larger amount of data than DINO [13] as shown in
the last row of Table III. Also, the overexposed areas and lane markings
create undesirable edges in the estimated disparity maps (can be observed in
Fig. 5, Fig. 6). This can be a limitation of the commonly used edge-aware
disparity smoothness loss $L_{\rm smooth}$. We currently leave these issues
for future investigation.

##  V Future Work and Conclusions

We introduce an algorithm achieving precise self-supervised stereo depth
estimation for nighttime conditions, leveraging visual foundation models. We
present an efficient masking method and distance regularizer to enhance the
accuracy of depth estimation, and novel, weighted evaluation metrics that
provide more accurate evaluation given the non-uniform ground truth depth
distributions. Our approach shows effective performance across a range of
challenging scenarios and generalizes well to unseen datasets.

##  VI Acknowledgments

This work was supported by AWS via the Oxford-Singapore Human-Machine
Collaboration Programme, and EPSRC via ACE-OPS (EP/S030832/1). The authors
would also like to thank the anonymous reviewers for their helpful comments.

## References

  * [1] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 3, pp. 1623–1637, 2020.
  * [2] R. Ranftl, A. Bochkovskiy, and V. Koltun, “Vision transformers for dense prediction,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 12 179–12 188.
  * [3] C. Godard, O. Mac Aodha, and G. J. Brostow, “Unsupervised monocular depth estimation with left-right consistency,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 270–279.
  * [4] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into self-supervised monocular depth estimation,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 3828–3838.
  * [5] Y. Wang, Y. Liang, H. Xu, S. Jiao, and H. Yu, “Sqldepth: Generalizable self-supervised fine-structured monocular depth estimation,” _arXiv preprint arXiv:2309.00526_ , 2023.
  * [6] R. Peng, R. Wang, Y. Lai, L. Tang, and Y. Cai, “Excavating the potential capacity of self-supervised monocular depth estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 15 560–15 569.
  * [7] K. Zhou, L. Hong, C. Chen, H. Xu, C. Ye, Q. Hu, and Z. Li, “Devnet: Self-supervised monocular depth learning via density volume construction,” in _European Conference on Computer Vision_. Springer, 2022, pp. 125–142.
  * [8] Z. Feng, L. Yang, L. Jing, H. Wang, Y. Tian, and B. Li, “Disentangling object motion and occlusion for unsupervised multi-frame monocular depth,” in _European Conference on Computer Vision_. Springer, 2022, pp. 228–244.
  * [9] Y. Wang, P. Wang, Z. Yang, C. Luo, Y. Yang, and W. Xu, “Unos: Unified unsupervised optical-flow and stereo-depth estimation by watching videos,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 8071–8081.
  * [10] G. Xu, X. Wang, X. Ding, and X. Yang, “Iterative geometry encoding volume for stereo matching,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 21 919–21 928.
  * [11] S. Gasperini, N. Morbitzer, H. Jung, N. Navab, and F. Tombari, “Robust monocular depth estimation under challenging conditions,” in _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2023.
  * [12] R. Garg, V. K. B.G., G. Carneiro, and I. Reid, “Unsupervised cnn for single view depth estimation: Geometry to the rescue,” in _Computer Vision – ECCV 2016_. Cham: Springer International Publishing, 2016, pp. 740–756.
  * [13] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021.
  * [14] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, “Dinov2: Learning robust visual features without supervision,” 2023.
  * [15] I. Fang, H.-C. Wen, C.-L. Hsu, P.-C. Jen, P.-Y. Chen, Y.-S. Chen _et al._ , “Es3net: Accurate and efficient edge-based self-supervised stereo matching network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 4471–4480.
  * [16] K. Saunders, G. Vogiatzis, and L. Manso, “Self-supervised monocular depth estimation: Let’s talk about the weather,” in _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2023.
  * [17] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 Year, 1000km: The Oxford RobotCar Dataset,” _The International Journal of Robotics Research (IJRR)_ , 2017.
  * [18] M. B. Vankadari, S. Garg, A. Majumder, S. Kumar, and A. Behera, “Unsupervised monocular depth estimation for night-time images using adversarial domain feature adaptation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:222133092
  * [19] C. Zhao, Y. Tang, and Q. Sun, “Unsupervised monocular depth estimation in highly complex environments,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , vol. 6, pp. 1237–1246, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:236469578
  * [20] L. Liu, X. Song, M. Wang, Y. Liu, and L. Zhang, “Self-supervised monocular depth estimation for all day images using domain separation,” in _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021.
  * [21] J. Spencer, R. Bowden, and S. Hadfield, “Defeat-net: General monocular depth via simultaneous unsupervised representation learning,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Los Alamitos, CA, USA: IEEE Computer Society, Jun 2020, pp. 14 390–14 401. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR42600.2020.01441
  * [22] K. Wang, Z. Zhang, Z. Yan, X. Li, B. Xu, J. Li, and J. Yang, “Regularizing nighttime weirdness: Efficient self-supervised monocular depth estimation in the dark,” in _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021.
  * [23] M. Vankadari, S. Golodetz, S. Garg, S. Shin, A. Markham, and N. Trigoni, “When the sun goes down: Repairing photometric losses for all-day depth estimation,” 2022.
  * [24] Y. Zheng, C. Zhong, P. Li, H. ang Gao, Y. Zheng, B. Jin, L. Wang, H. Zhao, G. Zhou, Q. Zhang, and D. Zhao, “Steps: Joint self-supervised nighttime image enhancement and depth estimation,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2023.
  * [25] H. Xu, J. Zhang, J. Cai, H. Rezatofighi, F. Yu, D. Tao, and A. Geiger, “Unifying flow, stereo and depth estimation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [26] Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_. Springer, 2020, pp. 402–419.
  * [27] J. Li, P. Wang, P. Xiong, T. Cai, Z. Yan, L. Yang, J. Liu, H. Fan, and S. Liu, “Practical stereo matching via cascaded recurrent network with adaptive correlation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 16 263–16 272.
  * [28] G. Xu, Y. Wang, J. Cheng, J. Tang, and X. Yang, “Accurate and efficient stereo matching via attention concatenation volume,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [29] A. Sharma, L.-F. Cheong, L. Heng, and R. T. Tan, “Nighttime stereo depth estimation using joint translation-stereo learning: Light effects and uninformative regions,” in _2020 International Conference on 3D Vision (3DV)_ , 2020, pp. 23–31.
  * [30] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, pp. 211–252, 2015.
  * [31] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel, “Deep vit features as dense visual descriptors,” _arXiv preprint arXiv:2112.05814_ , vol. 2, no. 3, p. 4, 2021.
  * [32] J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou, “Loftr: Detector-free local feature matching with transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 8922–8931.
  * [33] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superglue: Learning feature matching with graph neural networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 4938–4947.
  * [34] D. Hernandez-Juarez, A. Chacón, A. Espinosa, D. Vázquez, J. C. Moure, and A. M. López, “Embedded real-time stereo estimation via semi-global matching on the gpu,” _Procedia Computer Science_ , vol. 80, pp. 143–153, 2016.
  * [35] A. Sablayrolles, M. Douze, C. Schmid, and H. Jégou, “Spreading vectors for similarity search,” _arXiv preprint arXiv:1806.03198_ , 2018.
  * [36] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [37] U. Shin, J. Park, and I. S. Kweon, “Deep depth estimation from thermal image,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 1043–1053.
  * [38] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _International Conference on Learning Representations (ICLR)_ , San Diega, CA, USA, 2015.
  * [39] D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” _Advances in neural information processing systems_ , vol. 27, 2014.


# Combined Data and Deep Learning Model Uncertainties: An Application to the
Measurement of Solid Fuel Regression Rate

Georgios Georgalis
Data Intensive Studies Center
Tufts University
Medford, MA 02155, USA
<EMAIL_ADDRESS>
&Kolos Retfalvi, Paul E. DesJardin
Department of Mechanical and Aerospace Engineering
University at Buffalo, The State University of New York,
Buffalo, NY 14260, USA
{kretfalvi<EMAIL_ADDRESS>
&Abani Patra
Data Intensive Studies Center and Department of Mathematics
Tufts University
Medford, MA 02155, USA
<EMAIL_ADDRESS>

###### Abstract

In complex physical process characterization, such as the measurement of the
regression rate for solid hybrid rocket fuels, where both the observation data
and the model used have uncertainties originating from multiple sources,
combining these in a systematic way for quantities of interest (QoI) remains a
challenge. In this paper, we present a forward propagation uncertainty
quantification (UQ) process to produce a probabilistic distribution for the
observed regression rate $\dot{r}$. We characterized two input data
uncertainty sources from the experiment (the distortion from the camera
$U_{c}$ and the non-zero angle fuel placement $U_{\gamma}$), the prediction
and model form uncertainty from the deep neural network ($U_{m}$), as well as
the variability from the manually segmented images used for training it
($U_{s}$). We conducted seven case studies on combinations of these
uncertainty sources with the model form uncertainty. The main contribution of
this paper is the investigation and inclusion of the experimental image data
uncertainties involved, and how to include them in a workflow when the QoI is
the result of multiple sequential processes.

_K_ eywords Uncertainty characterization, Deep learning model uncertainty,
Image data uncertainty, Combustion Experiments, Regression rate density
estimation

##  1 Introduction

When analyzing a physical system, we use the “input data” (knowledge we have)
to run the “model” and evaluate its outputs (knowledge we want to obtain). All
of the components involved in the process carry their own assumptions,
limitations, and uncertainties. When the outputs are part of a larger
computational framework or are used in decision making, reporting the
associated uncertainty is required, because the modeling process and the model
evaluations are approximations of the underlying true physical process.
Probability models are common representations of such uncertainty in both the
inputs and outputs. Forward-propagation of the uncertainty of input data and
estimation of the probability distribution of the simulation output via
sampling based Monte-Carlo methods or functional approximations is abundant in
literature with recent applications in many fields (e.g., see [1], [2], [3]).
Similarly, observation data-driven calibration procedures using Markov Chain
Monte Carlo or variants are common [4].

However, in cases where the “input data” are results of a complex physical
process that is inherently variable (e.g., from a complex experiment), and the
“modeling” includes multiple sequential models and dependencies, propagating
and combining all uncertainty sources in a systematic way for a required
quantity of interest (QoI) remains a challenge [5]. This is especially true
when using operations like image segmentation models [6] for identification of
surfaces and interfaces which implicitly and explicitly use modeling and
calibration. In particular, we focus here on the characterization of the
“input data" uncertainty when it is acquired in a complex experiment and used
within a model-based interpretation that transforms the raw observation into a
QoI.

In hybrid rocket motor combustion, one QoI is the rate at which the fuel
surface recedes during the burn, defined as the regression rate $\dot{r}$. The
regression rate has direct impact on the geometrical design of the rocket
motor and its performance ([7], [8]). Due to the high cost of building a full-
scale hybrid motor, it is common practice to estimate the regression rate a
priori via a smaller-scale experiment of a slab burner [9]. Tracking the
surface of the fuel from experimental images during the burn with time data is
one way to estimate $\dot{r}$ [10]. In this case, “inputs” are all of the
assumptions and limitations of the experiment, the equipment, and any
processing to get the fuel surface images. “Modeling” is the process of
translating the images to the regression rate, which is practically a sequence
of multiple models: a deep learning convolutional network model to segment the
fuel masks of an entire experiment (e.g., Monte-Carlo Dropout (MCD) U-net from
[11]), a process to detect the boundaries of the segmented fuel masks, and a
process to estimate $\dot{r}$ from the changing boundaries with time data
(Fig. 1).

The regression rate and its measure of uncertainty are necessary to either
validate multi-scale computational models (e.g., simulations of turbulent
combustion chemistry coupled with flow dynamics [12]), or for decision making
on the rocket motor design and performance. In these coupled systems, there is
the additional requirement that the uncertainty information of the QoI be
expressed in a simple way (e.g., as a probability distribution) because
simulations can be sequential: the distribution of the regression rate
represented by an ensemble is an input to another simulation (e.g., see [13]
for an application of sequential uncertainty quantification in combustion
systems).

Figure 1: Overview of the workflow and uncertainties at each step to estimate
the fuel regression rate $\dot{r}$ from a slab burner experiment. Hybrid
rocket image (left) is NASA’s Peregrine rocket motor.

iterature has focused on separating the different types of uncertainty within
a model as aleatoric (AU) or epistemic uncertainty (EU) and how to measure
them in physical models such as hypersonic flows [14], in machine learning
[15], in Bayesian convolutional networks [16], and in safety decision making
[17] [18] among others. In the context of a complex “input data” and
“modeling” process as in the regression rate case (Fig. 1), these two distinct
uncertainty definitions are often challenging to separate or measure. For
example, the inherent camera structure and how it represents an image captured
in the experimental environment and whether the researcher misplaced the fuel
specimen by a few degrees include some inherent randomness, but we have the
capability to estimate some parts of that experimental randomness by analyzing
the setup and the optical errors. Therefore, in this work, we do not classify
uncertainty as aleatoric or epistemic, but rather characterize and quantify
the individual uncertainty measures based on their source (i.e., experiment or
modeling).

In this paper, we present an uncertainty quantification (UQ) process to
produce a probabilistic distribution for the regression rate $\dot{r}$ from
the input experimental images, by combining the input data uncertainty with
inference model uncertainty. We characterize the input data uncertainties from
the experiment $U_{data}=\\{U_{c},U_{\gamma}\\}$, the model form uncertainty
from the MCD U-net ($U_{m}$), and the variance in the model prediction from
the variability of manually tracing the masks used to train the MCD U-net
($U_{s}$). The process is shown in Fig. 1: (a) We introduce the input data
uncertainty $U_{data}$ to create sequenced ensembles of experimental images,
(b) pass the ensembles through the U-net to get an estimate of the mean
predicted masks $\mu_{mask}$ and corresponding uncertainty maps $U_{mask}$
which carry both the data and model uncertainties, and lastly (c) process each
sequenced ensemble of predicted fuel masks and uncertainty maps to get an
estimate for the regression rate $\dot{r}$ and its bounds.

The sources of input data uncertainty are most relevant for this problem
because optical distortion from the camera ($U_{c}$) or non-zero angle
placement of the fuel ($U_{\gamma}$) may misrepresent the fuel boundaries in
the images that are tracked to accurately measure the regression rate. The
model form uncertainty ($U_{m}$) is computed as the entropy of the probability
prediction vector of the MCD U-net. The possible variation in the manually
segmented masks used for training the MCD U-net is added as a prediction
variance $U_{s}$ directly to the final resulting uncertainty map $U_{mask}$.

The paper is organized as follows. Section 2 includes details about the
experimental setup and the characterization for each of the studied
uncertainty measures. Section 3 presents a series of cases that investigate
how the different types of uncertainties impact the overall flow of
uncertainty measure in the forward-propagation process (i.e., how the input
data uncertainties individually and together combine with the model form
uncertainty and the manual segmentation variance to form the output
uncertainty map used in the measuring $\dot{r}$). Section 4 is an application
of the entire workflow shown in Fig. 1, and includes the resulting
distribution for $\dot{r}$ via forward-propagation after including all the
uncertainty sources in sequence. Section 5 summarizes the conclusions of the
paper with directions on future work.

##  2 Formulation of Uncertainty Sources and Regression Rate Measurement

In this section, we describe the experimental setup and equipment used, the
characterization of the input data uncertainties
$U_{data}=\\{U_{c},U_{\gamma}\\}$ and how they are introduced to the
experimental images. $U_{c}$ corresponds to error from optical distortion that
is inherent to the camera placement in respect to the fuel specimen.
$U_{\gamma}$ corresponds to the non-zero angle fuel placement error, which may
result from the researcher not perfectly placing the fuel orthogonal to the
camera axis, thus having the camera misrepresent the distance between the fuel
boundaries correctly. We also show the formulation for the model form
uncertainty $U_{m}$ and the manual segmentation variance $U_{s}$, as well as
how the model outputs $\mu_{mask},U_{mask}$ are used to measure the regression
rate $\dot{r}$ and its bounds.

###  2.1 Experimental setup and its uncertainties

The experimental setup follows closely the one developed by Dunn et al. [19]
and can be seen in Fig. 2. The fuel specimen is placed in a chamber consisting
of two stainless steel plates on the top and bottom and high temperature
borosilicate glasses on the side of the experiment for optical access. The
chamber is 15.24cm long, 2.54cm tall and 2.54cm wide, with a oxidizer inlet
pipe of 1.83cm long and 2.54cm in diameter. Based on the entrance length, the
inlet flow is assumed to be fully developed as it enters the chamber. We used
lab-grade paraffin wax from the Carolina Biological Company as the fuel during
the experiments and they were cast in a stainless steel mold. The temperature
of the mold was monitored during the solidification process to avoid
impurities in the samples. The average dimensions for the fuel specimens used
in the experiments were 9.4mm wide, 80.7mm long and 11.2mm in height. Each
sample had a 45∘ slant in the front to guide the flow. The oxidizer used for
the experiment was 100$\%$ gaseous oxygen which was regulated with solenoids
and measured with an Omega FMA 1744a mass flow meter. The measurement range of
the flowmeter was 5 - 500 SLMs with $\pm 1.5\%$ accuracy.

To obtain the image dataset used in this paper, we conducted two experiments
with measured oxidizer mass fluxes of $G_{1}=6.96\frac{kg}{m^{2}s}$ and
$G_{2}=10.96\frac{kg}{m^{2}s}$ respectively. During the automated experimental
sequence the oxidizer was first introduced and the slab ignited using a Bosch
diesel glow plug which was lowered to make contact with the slab and later
retracted using a stepper motor. The experiment continued until the flame was
extinguished. The experiment was recorded with a Chronos 2.1 high speed camera
during the experimental runs, with a frame rate of 1000 frames per second. The
lens used was a NIKON AF Nikkor 50mm lens with apertures ranging from 1.8 to 8
with an exposure of 1 $\mu s$. The full burn of the specimen took between 8-12
seconds. Some example images from the experiment with oxidizer flux
$G_{2}=10.96\frac{kg}{m^{s}s}$ at different times can be seen in Fig. 3.

Figure 2: Slab burner test chamber with paraffin wax sample. Figure 3: Example
images from the experiment with oxidizer flux $G_{2}=10.96\frac{kg}{m^{s}s}$
at early (a), middle (b), and later (c) stages.

####  2.1.1 Optical distortion error $U_{c}$

Table 1: Camera resolution and parameter estimation from calibration Camera Parameter |
---|---
Resolution | 1920x1080 [pix]
$f_{x}$ | 4386.76 $\pm$ 27.41 [pix]
$f_{y}$ | 4374.34 $\pm$ 26.68[pix]
$s$ | 0.0 (perpendicular axes)
$c_{x}$ | 971.65 $\pm$ 1.25 [pix]
$c_{y}$ | 503.64 $\pm$ 1.22 [pix]
$k_{1}$ | 4.30 $\pm$ 0.24 [–]
$k_{2}$ | 28.74 $\pm$ 13.40 [–]

The image data from the experiment are not exactly accurate representations of
the real phenomena during the burn. One reason is that despite best efforts to
set the camera correctly for best image quality, the fuel surface moves with
respect to the camera during the experiment, adding a possibility of optical
distortions. To account for the possible camera distortions as part of the
input data uncertainty, we followed a calibration process by using
checkerboard patterns. Assuming the pinhole model for the camera, i.e. the
real points of the captured space are imaged by rays that pass through a
single origin of the camera lens [20], we estimated the intrinsic and
extrinsic parameters of the camera and the distortion coefficients as outlined
in [21].

Given the global coordinate system $(x,y,z)$ and the camera coordinate system
$(i,j,k)$, a point in the real world is represented by the position vector
$\mathbf{p}=[p_{x},p_{y},p_{z}]^{T}$. The same point in the camera coordinate
system is represented by the position vector
$\mathbf{q}=[q_{i},q_{j},q_{k}]^{T}=R^{T}(\mathbf{p}-\bm{\tau})$, where $R$ is
the rotation matrix and $\bm{\tau}$ the translation vector (extrinsic
parameters). The point represented in the camera coordinates as $\mathbf{q}$
is then mapped into the image plane using the intrinsic parameters matrix $K$
and represented as $\bm{r}=K\mathbf{q}=\begin{bmatrix}f_{x}&0&0\\\
s&f_{y}&0\\\ c_{x}&c_{y}&1\end{bmatrix}[q_{i},q_{j},q_{k}]^{T}$ where
$f_{x},f_{y}$ are the focal lengths, $s$ is the skew coefficient, and
$c_{x},c_{y}$ are the coordinates of the optical center. The rotation matrix
$R$, the translation vector $\bm{\tau}$, and the intrinsic matrix $K$ are all
estimated during the camera calibration process. In addition, we also estimate
the camera distortion coefficients $k_{1},k_{2}$ that characterize radial
distortion in the image coordinates [22]:
$r_{x,distorted}=\frac{r_{x}}{f_{x}}(1+k_{1}d^{2}+k_{2}d^{4})$ and
$r_{y,distorted}=\frac{r_{y}}{f_{y}}(1+k_{1}d^{2}+k_{2}d^{4})$, where
$d=(\frac{r_{x}}{f_{x}})^{2}+(\frac{r_{y}}{f_{y}})^{2}$.

To calibrate the camera, we took ten images of a square checkerboard pattern
with a side of 2 cm under different angles (Fig. 5), and processed them using
MATLAB’s camera calibrator app [23]. The resulting estimated parameters and
camera resolution are shown in Table 1. The rotation matrices and translation
vectors are not shown because they are different for each calibration image.
During the calibration process, the true locations of the checkerboard pattern
are detected and compared with their locations on the reconstructed images
from the camera model. The reconstruction error is then
$e_{i,j}=s_{i,j}-\hat{s_{i,j}}$, where $s_{i,j}$ are the true detected points
on the calibration patterns and $\hat{s_{i,j}}$ the re-constructed points. To
arrive at a measure of distortion uncertainty from the camera that can be used
to introduce uncertainty to the slab burner images, we count how many of the
calibration points have a total error that is greater than 1 pixel in
distance, for each of the calibration images. The threshold is set at 1 pixel
because when we represent the experimental images as tensors for the deep
learning model, an error greater than 1 pixel means the image point is
distorted enough to not populate that field in the tensor anymore. The total
number of distorted pixels from each calibration image are expressed as a
percentage of the total pixels in the image. The result of this process is
shown in Fig. 5: the expected maximum number of distorted points with an error
greater than 1 pixel corresponds to 0.7 $\%$ of the image and the minimum
number to $0.133\%$ of the image. Therefore, based on the calibration results,
we expect that an experimental image taken with our camera has a number of
distorted pixels $U_{c}\sim\mathcal{U}(0.133,0.7)[\%]$, expressed as a
percentage of the image. Since there are only two options for the pixels,
distorted or not, for a given experimental image $X_{k}$ of resolution 512 by
64, there is a corresponding distortion map $D_{map,k}$. The distortion map
follows the binomial distribution $D_{map,k}\sim B(n=64*512,p=U_{c})$. The
pixels in $X_{k}$ that have a success trial on the distortion map $D_{map,k}$,
are distorted, and therefore have their intensity reduced to zero when we
introduce distortion uncertainty $U_{c}$ to the original image (Alg. 1).

Given experimental image $X_{k}:\mathbb{P}_{i,j}=\rightarrow[0,255]^{3}$ with
$i=[0;512]$ and $j=[0;64]$

Step 1. Draw a sample for the percentage of distorted pixels
$U_{c}\sim\mathcal{U}(0.133,0.7)[\%]$

Step 2. Create the corresponding distortion map $D_{map,k}\sim
B(n=64*512,p=U_{c})$

Step 3. Create the uncertain image copy
$\widetilde{X_{k}}=\begin{cases}0&\text{for }D_{map,k}=1\\\
X_{k}&\text{otherwise}\end{cases}$

Algorithm 1 Add distortion uncertainty $U_{c}$ to experimental images

Figure 4: Relative positioning between the camera and ten 2cm-side checkboard
calibration images.

Figure 5: For each of the calibration images, we expect between 0.133 to 0.7
$\%$ of the image to be distorted for a distance greater than 1 pixel. From
this result, we assume a random experimental image from our camera to have a
number of distorted pixels $U_{c}\sim\mathcal{U}(0.133,0.7)[\%]$.

####  2.1.2 Non-zero angle fuel placement error $U_{\gamma}$

Under ideal circumstances, the slab burner is positioned perfectly orthogonal
to the axis of the camera. However, the researcher that places the fuel
specimen may not always do so with precision, and place the fuel at an offset
small angle $\gamma\neq 0$, which then adds uncertainty to the experimental
images via over- or under-estimating the distances between borders for the
specimen [24]. The relative border location is important in accurate
estimation of the regression rate, since we track the border changes to
measure it. The schematic in Fig. 7 shows the geometry of this phenomenon.
Given the true length of the fuel specimen $y=8.069[cm]$, the length of the
specimen to the camera $L=37.46[cm]$, the angle of the camera to the end of
the specimen if placed correctly $\beta=tan^{-1}\frac{y}{2L}=6.147^{o}$, and
solving the geometry, we get an estimate for the perceived specimen length
$y_{m}=y(cos\gamma+tan\beta sin\gamma)$. We express the error in the perceived
fuel length, caused by the non-zero angle placement of the specimen, as
$U_{\gamma}=\frac{y_{m}-y}{y}[\%]$. The extreme values for $U_{\gamma}$ are
directly related to the extreme values for the angle $\gamma$. We assume that
$-5^{o}\leq\gamma\leq 5^{o}$, because a placement by a larger angle would
likely be visually noticed by the researcher and corrected before the
experiment, which corresponds to $-1.319\leq U_{\gamma}[\%]\leq 0.558$ as
shown in Fig. 7. Given that we have no prior knowledge about the distribution
of the error, we assume the non-zero angle placement uncertainty to be uniform
$U_{\gamma}\sim\mathcal{U}(-1.319,0.558)[\%]$.

Figure 6: If the fuel specimen is not placed orthogonally to the camera, but
with an offset small angle $\gamma$, the distance between the fuel boundaries
is misrepresented ($y_{m}\neq y$).

Figure 7: Change of non-zero angle placement error $U_{\gamma}$ for different
placement angles $\gamma$. For this analysis, we choose $-5^{o}\leq\gamma\leq
5^{o}$ to be a possible range of angle values, which corresponds to
$-1.319\leq U_{\gamma}[\%]\leq 0.558$.

To introduce $U_{\gamma}$ to a given experimental image $X_{k}$, we first
estimate the pixel density for the fuel specimen $\rho_{k}$. Given the area of
the fuel surface (which can be taken from the manually segmented or predicted
mask) and the length of the slab, we estimate the density
$\rho_{k}=\frac{\text{fuel length in pixels}}{y}$ [pix/cm] for that image. We
then translate $U_{\gamma}$ into [cm] units $U_{\gamma,cm}=U_{\gamma}/100*y$,
expressing the uncertainty as the length-wise error between the boundaries.
Then the quantity $U_{\gamma,cm}*\rho_{k}$ provides the number of pixels that
the right fuel boundary has to shift to introduce the sampled value of
$U_{\gamma}$. If $U_{\gamma}>0$ then the right boundary is shifted to the
right, and if $U_{\gamma}<0$ then the right boundary is shifted to the left.
The process of shifting includes either removing a vertical strip in the
middle of the image and stitching the remaining image together adding
background pixels to the right ($U_{\gamma}<0$) or duplicating the vertical
strip in the middle of the image and removing some of the pre-existing
background on the right ($U_{\gamma}>0$). The manipulation happens in the
horizontal middle (at 256 pixels) since the right and left boundaries are far
away from that location. The reader is referred to Alg. 2 for the process of
introducing $U_{\gamma}$ to the images and two visual examples of the extreme
cases in Fig. 8.

Given experimental image $X_{k}:\mathbb{P}_{i,j}=\rightarrow[0,255]^{3}$ with
$i=[0;512]$ and $j=[0;64]$

Step 1. Draw a sample for the non-zero angle uncertainty
$U_{\gamma}\sim\mathcal{U}(-1.319,0.558)[\%]$

Step 2. Translate to length-wise units and find pixel density
$U_{\gamma,cm}=U_{\gamma}y/100$ and $\rho_{k}=\frac{\text{fuel length in
pixels}}{y}$

Step 3. If $U_{\gamma}>0$:
$\begin{cases}\text{Loc1}=\text{floor}(256-U_{\gamma,cm}\rho_{k}/2)\\\
\text{Loc2}=\text{floor}(256+U_{\gamma,cm}\rho_{k}/2)\\\ \text{Added
slice}\hskip 10.00002ptS_{k}=X_{k}[\text{loc1}:\text{loc2}]\\\
\widetilde{X_{k}}=\text{append}([X_{k}[:256],S_{k},X_{k}[256:512-\text{size}(S_{k})])\\\
\end{cases}$

If $U_{\gamma}<0$:
$\begin{cases}\text{Loc1}=\text{floor}(256-U_{\gamma,cm}\rho_{k}/2)\\\
\text{Loc2}=\text{floor}(256+U_{\gamma,cm}\rho_{k}/2)\\\
\widetilde{X_{k}}=\text{append}([X_{k}[:\text{loc1}],X_{k}[\text{loc2}:512],[0]_{\text{loc2-loc1}})\\\
\end{cases}$

Algorithm 2 Add non-zero angle placement uncertainty $U_{\gamma}$ to
experimental images Figure 8: Different versions of the same experimental
image at extreme values: (a) $U_{\gamma}=0.558\%$, (b) $U_{\gamma}=0$, and (c)
$U_{\gamma}=-1.319\%$. The red line is a reference point on the edge of the
right boundary in the image with no uncertainty to aid the visualization of
the difference in the cases with uncertainty present.

###  2.2 Monte-Carlo Dropout (MCD) U-net for Segmentation

####  2.2.1 Model Form Uncertainty $U_{m}$

We created the U-net architecture using the Keras library [25]. Fig. 9 shows a
schematic of the U-net architecture with five dropout layers. The U-net has an
encoding path that initially extracts the most important features from the
image as resolution reduces and depth increases, followed by a decoding path
to return to higher resolutions by reconstructing the image using up-sampling
[26]. The encoding path consists of four sequences of: two 3x3 convolutions,
batch normalization, ReLU activation, and max pooling. The decoding path
consists of four sequences of: 3x3 up-convolutions with a concatenation of the
feature map from the corresponding level in the encoding path, two 3x3
convolutions, batch normalization, and lastly a ReLU activation. Finally, the
U-net includes a 1x1 convolution operation to output the segmented fuel mask.
The total number of trainable parameters in the network are 492,609. The
parameters of a convolution layer are equal to $out\times[in\times(3\times
3)+1]$, where out is the number of features after the convolution, in is the
number of features before the convolution, the 9 parameters of the $3\times 3$
filter, and one parameter at each node for the bias term. For example, the
first convolution layer takes the original image ($64\times 512\times 3$),
which after the convolution becomes ($64\times 512\times 8$), therefore that
layer has $8\times[3\times(3\times 3)+1]=224$ parameters. Each of the batch
normalization layers adds 4 additional parameters per feature of the previous
layer [27]. For example, since the output after the first convolution layer
has 8 features, the corresponding batch normalization layer will have 32
parameters (16 of which are non-trainable).

Figure 9: The U-net neural network architecture with dropout in the
intermediate layers. Given an input experimental image, the U-net outputs the
predicted fuel segmentation mask.

To estimate the prediction model form uncertainty $U_{m}$, we implemented
Monte-Carlo Dropout (MCD), a Bayesian approximation in deep learning models
[28]. The dropout process randomly silences neurons with a probability of
$p_{D}$ in the intermediate layers of the U-net. Then, at inference, the model
is sampled with dropout which is an approximation to sampling from the
posterior weight distribution of a fully Bayesian network [29]. Given a new
test image $X_{k}$, and the training set $(X_{train},Y_{train})$, the
resulting mean segmented mask can be approximated using Monte-Carlo
integration as:

$\hat{p}(y=c|X_{k,test},X_{train},Y_{train})\approx\frac{1}{T}\sum_{t=1}^{T}\sigma(f^{\hat{W_{t}}}(X_{k}))$
(1)

where $\hat{p}(y)$ corresponds to the predicted probability that a pixel in
the test image $X_{k}$ is classified as $c=\\{\text{Fuel}\\}$, $T$ is the
number samples for the weights set to 20 [30], $\sigma(\cdot)$ is the softmax
function, and $f$ is the model output given the weights $\hat{W_{t}}\sim
q_{\theta}(W_{t})$, sampled from the dropout approximating variational
distribution $q_{\theta}(W_{t})$ [16], which is defined for every layer $t$:

$W_{t}=w_{t}\cdot diag([z_{t,m}]_{m=1}^{K_{t}}])$ (2)

$\displaystyle z_{t,m}\begin{cases}\sim Bernoulli(p_{D})\hskip 5.0pt&\text{for
layers }t=7,10,13,15,18\hskip 5.0pt\text{and}\hskip 5.0ptm=1,...,K_{t-1}\\\
=1&\text{otherwise}\end{cases}$ (3)

where $w_{t}$ are the weights of the trained network. The diag($\cdot$)
operator maps vectors to diagonal matrices whose diagonals are the elements of
the vectors [31]. The variable $z_{t,m}=0$ corresponds to node $m$ in layer
$t-1$ to be dropped out as an input to the next layer, whereas $z_{t,m}=1$
corresponds to the trained weights $w_{t}$ to be used as is, which is also the
case for layers without dropout, $p_{D}=0.5$ is the dropout probability,
$K_{t}$ is the number of nodes/units within layer $t$. We note that the
dropout probability is non-zero only in the intermediate blocks of the U-net,
which corresponds to layers 7, 10, 13, 15, 18 of the network.

The model form uncertainty for the predicted probability $\hat{p}$ can be
expressed as the corresponding map of the cross-entropy of the two class
dimensions [32, 33, 16]:

$U_{m}=-\sum_{c=1}^{2}\hat{p}_{c}\log(\hat{p}_{c})=-[\hat{p}_{{}_{fuel}}\log(\hat{p}_{{}_{fuel}})+\hat{p}_{{}_{background}}\log(\hat{p}_{{}_{background}})]$
(4)

where $U_{m}$ is the model form uncertainty map, $\hat{p}_{{}_{fuel}}$ is the
predicted probability that a given pixel corresponds to fuel, and
$\hat{p}_{{}_{background}}$is the predicted probability that a given pixel
corresponds to noise/background.

The U-net was trained for 79 epochs with 179 images from the early, mid, and
late phases from the first experiment with oxidizer flux
$G_{1}=6.96\frac{kg}{m^{2}s}$, accompanied with their manually segmented fuel
masks that were traced using VGG [34]. Fig. 10 shows the loss progression
during training. Observing the results from the testing set (Fig. 11), the
U-net predicts the segmentation masks correctly, even for highly saturated
images and without including any other components of the chamber as part of
the mask (e.g., the glow plug). The uncertainty maps show the largest level of
uncertainty to be around the fuel boundary geometries. The model form
uncertainty further inside and outside of the identified fuel boundary is very
small.

Figure 10: Binary cross-entropy loss during the training of the U-net used for
this work. Some abrupt changes in the loss are due to dropout effects. The
training stops once the validation loss has not improved after 20 consecutive
epochs. Figure 11: Results from the baseline U-net model without any data
uncertainty introduced to the images. The U-net performs well on all images,
including low, medium, and high saturation levels during the phases of the
experiment.

####  2.2.2 Manual segmentation uncertainty $U_{s}$

In the process of training the U-net, we inherently make the assumption that
the manually segmented fuel masks are ground truth. However, there is expected
to be some variability between individuals that segment the fuel masks, and
repeating the process multiple times to investigate the level of this
variability are beyond the research scope of this paper. There have been
studies to quantify the effect of improperly segmented fuel masks to the U-net
outputs (e.g., see [35], [36]), which found that the error in the segmentation
accuracy for the MNIST [37] and MSLesion [38] dataset is 0.83 $\%$ and 0.56
$\%$ respectively. The MNIST dataset consists of grayscale images of digits
from 0 to 9, and the MSLesion dataset of 21 medical scans. Given that the fuel
segmentation masks are simple with only two labels, and without any prior
knowledge, we assume the manual segmentation prediction uncertainty to be
similar to those datasets and uniform $U_{s}\sim\mathcal{U}(0.56,0.83)\%$,
expressed as a percentage variance of the corresponding segmentation
probability $\hat{p}_{{}_{fuel}}$. Therefore, the complete resulting
uncertainty map for the mean prediction $\mu_{mask}$ of the U-net after
considering all the uncertainty sources is

$U_{mask}=U_{m}|(U_{c},U_{\gamma})+\hat{p}_{{}_{fuel}}U_{s}$ (5)

where $U_{mask}$ is the complete resulting uncertainty map after all sources
of uncertainty are included, $U_{m}$ is the model form uncertainty, $U_{c}$ is
the distortion uncertainty, $U_{\gamma}$ is the non-zero angle placement
uncertainty, $\hat{p}_{{}_{fuel}}$ is the corresponding segmentation
probability for the fuel class, and $U_{s}$ is the manual segmentation
prediction uncertainty.

###  2.3 Fuel surface detection to measure $\dot{r}$

To approximate the regression rate from the binary fuel masks, we follow the
approach described in [10]. For a given sequence of fuel masks $\mu_{mask,k}$
representing the fuel profile between time intervals $\Delta t$, we track the
height of the top surface of the fuel profile through calibration points. At
each point, the localized regression rate is

$\dot{r_{l,k}}=\frac{h(t_{k+1})-h(t_{k})}{\Delta t}$ (6)

where $h(\cdot)$ is the fuel height for a given time measured from the bottom
of the fuel mask and $\Delta t$ is the time difference between two frames. The
total regression rate for the experiment is then the average localized
regression rate from all points:

$\dot{r}=\frac{\sum_{k=1}^{M}\dot{r_{l,k}}}{M}$ (7)

Given that each of the predicted fuel masks $\mu_{mask,k}$ also has an
accompanying uncertainty map $U_{mask,k}$ after the inclusion of all
uncertainties, we calculate two extreme values for the total regression rate
estimate $\dot{r}$. The upper bound is found by adding the uncertainty maps
$U_{mask,k}$ to the corresponding fuel masks $\mu_{mask,k}$, flipping the
class from “background” to “fuel” for any pixels that now have
$\hat{p}_{fuel}\geq 0.5$ with the added uncertainty, and then re-calculating a
total regression rate estimate $\dot{r}^{+}$. The lower bound comes from the
opposite process: subtracting the uncertainty maps $U_{mask,k}$ from the
corresponding fuel masks $\mu_{mask,k}$, flipping the class from “fuel” to
“background” for any pixels that now have $\hat{p}_{fuel}<0.5$ after the
subtracted uncertainty, and re-calculating a total regression rate estimate
$\dot{r}^{-}$. From this entire process, for a given sequence of experimental
images (with or without added data uncertainties $U_{c},U_{\gamma}$) and using
the U-net (with or without the manual segmentation uncertainty $U_{s}$), we
calculate a mean estimate for the total regression rate from the experiment
$\dot{r}$ with its bounds $(\dot{r}^{-},\dot{r}^{+})$.

##  3 Combination and Forward-propagation of uncertainty sources

In this section, we study all possible combinations of how each of the
individual data uncertainty sources shown in Fig. 1 impact the overall
uncertainty propagation process. For each of the three uncertainty sources
$U_{c},U_{\gamma},U_{s}$, we consider two scenarios: either they do not
contribute (i.e., are zero) or do contribute (i.e., are uniformly distributed
as described in the previous section). All possible cases are shown in the
Table 2.

Table 2: Studied cases of uncertainty sources Case $\\#$ | Distortion $U_{c}$ | Non-zero angle $U_{\gamma}$ | Manual segmentation $U_{s}$
---|---|---|---
Baseline | 0 | 0 | 0
1 | $\mathcal{U}(0.133,0.7)\%$ | 0 | 0
2 | 0 | $\mathcal{U}(-1.319,0.558)\%$ | 0
3 | $\mathcal{U}(0.133,0.7)\%$ | $\mathcal{U}(-1.319,0.558)\%$ | 0
4 | 0 | 0 | $\mathcal{U}(0.56,0.83)\%$
5 | $\mathcal{U}(0.133,0.7)\%$ | 0 | $\mathcal{U}(0.56,0.83)\%$
6 | 0 | $\mathcal{U}(-1.319,0.558)\%$ | $\mathcal{U}(0.56,0.83)\%$
7 | $\mathcal{U}(0.133,0.7)\%$ | $\mathcal{U}(-1.319,0.558)\%$ | $\mathcal{U}(0.56,0.83)\%$

First, we observe how the error from the camera distortion $U_{c}$ and the
non-zero angle placement error $U_{\gamma}$ impact the U-net model form
uncertainty map $U_{m}$ separately and together (cases 1,2, and 3). Secondly,
we observe how the manual segmentation error $U_{s}$ changes the complete
uncertainty map $U_{mask}$ individually and together with $U_{c},U_{\gamma}$
(cases 4, 5, and 6). Lastly, we showcase an example where all three
uncertainties are included (case 7), which is representative of the approach
followed in the calculation of the full probability distribution of the
regression rate in the next section, which is the end goal of the paper. For
these studies, we used three representative experimental images from the
testing set of the U-net that correspond to three different image saturation
levels in the experiment: ”low“, ”mid“, and ”high“, as shown in Fig. 12. The
histograms are the model form uncertainty near the boundary for each image.
Observing the model form uncertainty near the fuel boundary, we identify three
different modes depending on the saturation level in the experimental images.
Images with low saturation ( (a) in Fig. 12), show an approximately uniform
profile across all uncertainty ranges. Images of medium saturation ( (b) in
Fig. 12) show a bi-modal behavior for $U_{m}$ with a uniform profile in
between the peaks. Lastly, images with high saturation ( (c) in in Fig. 12)
has again a bi-modal profile similar to the medium saturation images, but the
intermediate levels of uncertainty are higher, which is attributed to the
higher saturation from the more violent burn.

Figure 12: Example images from the experiment with different saturation levels
at low (a), medium (b), and high (c) with their corresponding model form
uncertainty histograms. We used these representative images to account for
different image qualities, since the uncertainty distribution has a different
profile for each of them.

We follow a standard forward-propagation uncertainty approach: If $U_{c}$ and
$U_{\gamma}$ are present, we create an ensemble of perturbed/uncertain copies
of a given image by following the process in Alg. 1 or Alg. 2 respectively,
and pass these uncertain copies through the U-net to get the model form
uncertainty map $U_{m}$. If $U_{s}$ is present, that is added to the model
form uncertainty map $U_{m}$ proportionally to the predicted probability of
each pixel (Eqn. 5). For each of these cases, we sampled the corresponding
uniform distributions $N=10,000$ times and show the resulting uncertainty
distribution compared to the baseline case where no uncertainties are present.

Figure 13: Case 1: Impact of the distorted pixel uncertainty $U_{c}$ to the
model form uncertainty $U_{m}$ for three levels of saturation for the
experimental images: low (a), medium (b), and high (c). For the image with
high saturation, distortion changes the uncertainty profile for the lower
uncertainty values.

For case 1, where only the uncertainty from the distorted pixels $U_{c}$ is
present, we find that distortion adds some variability to the $U_{m}$
probability distribution, but still follows the overall pattern of the
distribution without distortion in the low and medium saturated images. The
small variability is expected since the distortion is added uniformly to the
entire image. However, in the highly saturated case, distortion appears to be
a moderating factor to the large peaks of small uncertainty in the original
image. The explanation for this irregularity is justified in Fig. 13. When
distortion is present in the highly saturated experimental image, the
uncertainty around distorted points with high density is much larger compared
to the image without distortion. The distorted points outside the fuel
boundary do not contribute to the uncertainty since they were already part of
the background. However, the distorted pixels that are present within the fuel
boundary, add additional uncertainty, which changes the bimodal profile for
the probability density present in the baseline case to a more uniform profile
for the lower uncertainty values.

Figure 14: Case 2: Impact of the non-zero angle fuel placement uncertainty
$U_{\gamma}$ to the model form uncertainty $U_{m}$ for three levels of
saturation for the experimental images: low (a), medium (b), and high (c). The
overall distribution of uncertainty is maintained.

For case 2, where only the non-zero angle fuel placement uncertainty
$U_{\gamma}$ is present, we find that some variability is added, but the
overall pattern of the distribution of uncertainty compared to the baseline
case is maintained for all saturation levels (Fig. 14). Case 3, which includes
both data uncertainties $U_{c},U_{\gamma}$, also appears to maintain the
overall profile, but the non-zero angle effects appear to negate some of the
variability caused by distortion, likely due to the removal or addition of
parts of the same image.

Figure 15: Case 3: Impact of the non-zero angle fuel placement uncertainty
$U_{\gamma}$ and distorted pixel uncertainty $U_{c}$ to the model form
uncertainty $U_{m}$ for three levels of saturation for the experimental
images: low (a), medium (b), and high (c). Compared to case 1, the presence of
non-zero angle uncertainty $U_{\gamma}$ negates some of the variability caused
by the distortion uncertainty $U_{c}$. Figure 16: Case 4: Impact of the manual
segmentation uncertainty $U_{s}$ to the overall uncertainty map $U_{mask}$ for
three levels of saturation for the experimental images: low (a), medium (b),
and high (c). The added effect of this uncertainty appears as a right-shift to
the uncertainty distribution profile.

Cases 4–6 show the effects of the manual segmentation uncertainty $U_{s}$
individually and together with the other uncertainty sources. $U_{s}$ has an
added effect based on the predicted probability of a given pixel, therefore
the additive effects are present for those pixels with high fuel probability
(i.e., within and on the fuel boundary). The result appears as a right-shift
to the uncertainty distribution profile (Figs. 16, 17, 18).

Figure 17: Case 5: Impact of the distorted pixel uncertainty $U_{c}$ and the
manual segmentation uncertainty $U_{s}$ to the overall uncertainty map
$U_{mask}$ for three levels of saturation for the experimental images: low
(a), medium (b), and high (c). Figure 18: Case 6: Impact of the non-zero angle
fuel placement uncertainty $U_{\gamma}$ and the manual segmentation
uncertainty $U_{s}$ to the overall uncertainty map $U_{mask}$ for three levels
of saturation for the experimental images: low (a), medium (b), and high (c).

Case 7 shows the combined effects of the studied uncertainties, that appear to
be mixed: a right-shift caused by the manual segmentation uncertainty $U_{s}$,
a moderating effect caused by removing or replicating parts of these images
from $U_{\gamma}$, and added variability (and profile change for the high-
saturated images) due to more areas with uncertainty from $U_{c}$ (Fig. 19).

Figure 19: Case 7: Impact of the distorted pixel uncertainty $U_{c}$, the non-
zero angle fuel placement uncertainty $U_{\gamma}$, and the manual
segmentation uncertainty $U_{s}$ to the overall uncertainty map $U_{mask}$ for
three levels of saturation for the experimental images: low (a), medium (b),
and high (c).

##  4 Estimation of the distribution of $\dot{{r}}$

The goal of this section is to implement the framework shown in Fig. 1 for the
experiment with oxidizer mass flux $G_{2}=10.96\frac{kg}{m^{2}s}$, and find
the probability distribution of the QoI: the total regression rate $\dot{r}$.
To begin, we first capture a sequence of images from the experiment at fixed
frame intervals (Fig. 20).

Figure 20: Images from experiment 2 with with oxidizer mass flux
$G_{2}=10.96\frac{kg}{m^{2}s}$. The frames shown are taken at a fixed interval
of 0.32 seconds. The sequence of the burn starts from the top left and ends at
the bottom right. The goal is to measure the probability distribution of the
average regression rate, the quantity that describes how fast the fuel recedes
during the burn. Figure 21: One ensemble of predicted fuel masks from
experiment 2 with with oxidizer mass flux $G_{2}=10.96\frac{kg}{m^{2}s}$, with
visible fuel tracing. The process of calibration points ensures that in cases
where the U-net misclassifies some part of the picture as fuel, which is
likely in the later parts of the experiment, the algorithm approximates the
boundary correctly for the regression rate calculation.

Then, we follow a forward-propagation Monte-Carlo approach: the data
uncertainties $U_{c},U_{\gamma}$ are uniformly sampled and added to the
sequence of images from experiment 2 (Fig. 20), the images are passed through
the U-net to get the segmented fuel masks $\mu_{mask,k}$ , the manual
segmentation uncertainty $U_{s}$ is uniformly sampled and added to the model
form uncertainty $U_{m}$ (Eqn. 5) to get the overall uncertainty maps
$U_{mask,k}$ for all the experimental images (see Fig. 21 for one example of
an ensemble of fuel masks with profile tracking). From one sequenced ensemble
of $\mu_{mask,k}$ and $U_{mask,k}$, one point measurement of the total
regression rate $\dot{r}$ is computed together with its bounds
($\dot{r}^{+},\dot{r}^{-})$. The process is repeated until the variance of the
final probability distribution of the regression rate $\sigma_{\dot{r}}^{2}$
converges to a tolerance of $e^{-6}$ (Alg. 3).

The distribution of the regression rate is shown in Fig. 22. We compare two
instances: a probability distribution if we consider the uncertainty maps
$U_{mask}$ that provide the bounds for the regression rate with each ensemble
or if we do not. The probability distribution considering only the mean masks
$\mu_{mask}$ from the ensembles resembles a Gaussian distribution with a mean
regression rate $\mu_{\dot{r_{\mu}}}=0.759954\frac{mm}{s}$. Considering the
uncertainty maps that provide the bounds for the regression rate from each
ensemble, adds variability to both tail-ends of the distribution with a shift
of the mean to $\mu_{\dot{r_{U}}}=0.740032\frac{mm}{s}$. In previous work
where we only considered the model form uncertainty $U_{m}$, we found the
regression rate for an experiment with $G=9.58\frac{kg}{m^{2}s}$ to be
$0.74\pm 0.09\frac{mm}{s}$. We note that when the uncertainty maps are
included in the calculation, a significant amount of density is shifted
towards the smaller regression rate values, because of how the surface
tracking algorithm works. The surface tracking algorithm accounts for a
physical constraint: it does not allow for the fuel surface to increase in
size during the burn, which would be a violation of combustion laws. Some
images in the later stages of the experiment ensembles can often result in
larger fuel surface than previous time steps when we add the uncertainty
masks, and that effect is neglected. Uncertainty masks tend to have higher
values as the burn proceeds (images are more saturated). In the subtracting
case, the equivalent scenario does not happen; the higher uncertainty maps in
the later stages of the burn will be considered normally. Therefore, the
process penalizes the higher regression rates when uncertainties are added,
resulting in a more significant density shift to lower values. The
contributions of this paper agree with trends from previous literature, but
add the additional step of characterization and quantification of data
uncertainty which translates to more accurate representation of the
distribution of the regression rate when measured from images.

Given a sequence of experimental images
$X_{k}:\mathbb{P}_{i,j}=\rightarrow[0,255]^{3}$ with $i=[0;512]$ and
$j=[0;64]$

While $\sigma_{\dot{r}}^{2,new}-\sigma_{\dot{r}}^{2,old}>e^{-6}$ do:

Step 1. Draw a sample for the non-zero angle uncertainty
$U_{\gamma}\sim\mathcal{U}(-1.319,0.558)[\%]$

Step 2. Draw a sample for the distortion uncertainty
$U_{c}\sim\mathcal{U}(0.133,0.7)[\%]$

Step 3. Introduce $U_{c},U_{\gamma}$ to the ensemble $X_{k}$ (Alg. 1 and Alg.
2)

Step 4. Pass the ensemble through U-net to get the mean predicted fuel masks
$\mu_{mask,k}$ and the mean model form uncertainty maps $U_{m,k}$.

Step 5. Draw a sample for the manual segmentation uncertainty
$U_{s}\sim\mathcal{U}(0.56,0.83)[\%]$

Step 6. Compute overall uncertainty map for the ensemble
$U_{mask,k}=U_{m,k}|(U_{c},U_{\gamma})+\hat{p}_{{}_{fuel,k}}U_{s}.$

Step 7. Compute point estimate for regression rate $\dot{r}$ and bounds
$\dot{r}^{+},\dot{r}^{-}$ from $\mu_{mask,k}$ and $U_{mask,k}$

Step 8. Append probability distribution
$p(\dot{r})=\mathrel{+}\\{\dot{r},\dot{r}^{+},\dot{r}^{-}\\}$ and update
variance $\sigma_{\dot{r}}^{2,new}$

End While

Algorithm 3 Estimation of the distribution of the regression rate $\dot{r}$
Figure 22: The probability distribution of the regression rate for the
experiment with oxidizer mass flux $G_{2}=10.96\frac{kg}{m^{2}s}$. The grey
curve only considers the mean fuel masks $\mu_{mask}$ for the calculation
(i.e., does not include the regression rate bounds using $U_{mask}$), whereas
the blue curve includes the information from the uncertainty maps. The mean of
the probability distribution from the mean masks only is
$\mu_{\dot{r_{\mu}}}=0.759954\frac{mm}{s}$ (grey curve) and considering the
uncertainty maps as well is $\mu_{\dot{r_{U}}}=0.740032\frac{mm}{s}$. The
uncertainty maps translate to signficant variability to both tail-ends of the
distribution.

##  5 Conclusions and Future Work

In this work, we presented a full forward-propagation framework for
uncertainty quantification of the regression rate from experimental images. We
characterized uncertainty sources that are pertinent to this case: distortion
uncertainty $U_{c}$ from the camera, non-zero angle placement of the fuel
$U_{\gamma}$, manual segmentation training uncertainty $U_{s}$, and model form
uncertainty from the Monte-Carlo Dropout U-net $U_{m}$. The image data
generated for this work came from two slab burner experiments with measured
oxidizer mass fluxes of $G_{1}=6.96\frac{kg}{m^{2}s}$ and
$G_{2}=10.96\frac{kg}{m^{2}s}$. We conducted seven case studies of all
possible combinations between the three uncertainty sources and found that
their effects combine: a right-shift in the model form uncertainty
distribution caused by the additive effect of the manual segmentation
uncertainty $U_{s}$, a moderating effect from $U_{\gamma}$ caused by the
effect of placing the fuel under a small angle, and the overall added
variability throughout the model form uncertainty distribution (and profile
change for the high-saturated images) due to more areas with uncertainty
caused from the distortion uncertainty $U_{c}$. We completed our study by
using the images from the experiment with oxidizer flux
$G_{2}=10.96\frac{kg}{m^{2}s}$, generating a large ensemble of their uncertain
copies by sampling the three uncertainty sources, and tracking the fuel
profiles in each case to estimate the full regression rate probability
distribution. The probability distribution considering only the mean masks
$\mu_{mask}$ from the ensembles resembles a Gaussian distribution with a mean
regression rate $\mu_{\dot{r_{\mu}}}=0.759954\frac{mm}{s}$. Considering the
uncertainty maps that provide the bounds for the regression rate from each
ensemble, adds variability to both tail-ends of the distribution with a shift
of the mean to $\mu_{\dot{r_{U}}}=0.740032\frac{mm}{s}$.

In future work, we will use the resulting probability distribution of the
regression rate from experimental data shown in this work, together with
experiments on other oxidizer fluxes, to validate an equivalent estimate for
the regression rate from coupled flow and chemistry simulations for the slab
burner combustion phenomena using our open-source software framework ABLATE
(https://ablate.dev/).

## Acknowledgments

The authors would like to thank the reviewer comments that greatly improved
the manuscript.

This work was supported by the United States Department of Energy’s (DoE)
National Nuclear Security Administration (NNSA) under the Predictive Science
Academic Alliance Program III (PSAAP III) at the University at Buffalo, under
contract number DE-NA0003961.

Computational resources: The authors acknowledge the Tufts University High
Performance Compute Cluster which was used for training the U-net models and
running the simulations for the uncertainty analysis.

## References

  * [1] Antonio M. Moreno-Rodenas, Franz Tscheikner-Gratl, Jeroen G. Langeveld, and Francois H.L.R. Clemens.  Uncertainty analysis in a large-scale water quality integrated catchment modelling study.  Water Research, 158:46–60, 2019.
  * [2] Jingye Tan, Umberto Villa, Nima Shamsaei, Shuai Shao, Hussein M. Zbib, and Danial Faghihi.  A predictive discrete-continuum multiscale model of plasticity with quantified uncertainty.  International Journal of Plasticity, 138:102935, 2021.
  * [3] Reese E. Jones, Michael T. Redle, Hemanth Kolla, and Julia A. Plews.  A minimally invasive, efficient method for propagation of full-field uncertainty in solid dynamics.  International Journal for Numerical Methods in Engineering, 122(23):6955–6983, 2021.
  * [4] Ralph Smith.  Uncertainty Quantification.  SIAM, 2014.
  * [5] Apostolos F Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis.  Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons, 2022.
  * [6] Michael C. Krygier, Tyler LaBonte, Carianne Martinez, Chance Norris, Krish Sharma, Lincoln N. Collins, Partha P. Mukherjee, and Scott A. Roberts.  Quantifying the unknown impact of segmentation uncertainty on image-based simulations.  Nature Communications, 12:5414, 2021.
  * [7] G. Zilliac and M. Karabeyoglu.  Hybrid rocket fuel regression rate data and modeling.  In 42nd AIAA/ASME/SAE/ASEE Joint Propulsion Conference and Exhibit, 2006.
  * [8] Greg Zilliac, George T. Story, Ashley C. Karp, Elizabeth T. Jens, and George Whittinghill.  Combustion efficiency in single port hybrid rocket engines.  In AIAA Propulsion and Energy 2020 Forum, 2020.
  * [9] M. Karabeyoglu, B. Cantwell, and D. Altman.  Development and testing of paraffin-based hybrid rocket fuels.  37th Joint Propulsion Conference and Exhibit, 2001.
  * [10] Kenneth Budzinski, Siddhant S. Aphale, Elektra Katz Ismael, Gabriel Surina, and Paul E. DesJardin.  Radiation heat transfer in ablating boundary layer combustion theory used for hybrid rocket motor analysis.  Combustion and Flame, 217:248–261, 2020.
  * [11] Gabriel Surina, Georgios Georgalis, Siddhant S. Aphale, Abani Patra, and Paul E. DesJardin.  Measurement of hybrid rocket solid fuel regression rate for a slab burner using deep learning.  Acta Astronautica, 190:160–175, 2022.
  * [12] Evatt R Hawkes, Ramanan Sankaran, James C Sutherland, and Jacqueline H Chen.  Direct numerical simulation of turbulent combustion: fundamental insights towards predictive models.  Journal of Physics: Conference Series, 16:65–79, jan 2005.
  * [13] Sajjad Yousefian, Gilles Bourque, and Rory F.D. Monaghan.  Bayesian inference and uncertainty quantification for hydrogen-enriched and lean-premixed combustion systems.  International Journal of Hydrogen Energy, 46(46):23927–23942, 2021\.
  * [14] Benjamin Bettis and Serhat Hosder.  Uncertainty quantification in hypersonic reentry flows due to aleatory and epistemic uncertainties.  In 49th AIAA Aerospace Sciences Meeting including the New Horizons Forum and Aerospace Exposition, 2011.
  * [15] Eyke Hüllermeier and Willem Waegeman.  Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods.  Machine Learning, 110(3):457–506, 2021.
  * [16] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In Advances in Neural Information Processing Systems, volume 30, 2017.
  * [17] Stephen C. Hora.  Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management.  Reliability Engineering and System Safety, 54(2):217–223, 1996\.
  * [18] Armen Der Kiureghian and Ove Ditlevsen.  Aleatory or epistemic? does it matter?  Structural Safety, 31(2):105–112, 2009.
  * [19] C. Dunn, G. Gustafson, J. Edwards, T. Dunbrack, and C. Johansen.  Spatially and temporally resolved regression rate measurements for the combustion of paraffin wax for hybrid rocket motor applications.  Aerospace Science and Technology, 72:371–379, 2018.
  * [20] J.Y. Bouguet and P. Perona.  Camera calibration from points and lines in dual-space geometry.  In Proceedings of the 5th European Conference on Computer Vision, pages 2–6, Jun 1998.
  * [21] R. Juarez-Salazar, J. Zheng, and Victor H. Diaz-Ramirez.  Distorted pinhole camera modeling and calibration.  Applied Optics, 59(36):11310–11318, 2020.
  * [22] J. Heikkila and O. Silven.  A four-step camera calibration procedure with implicit image correction.  In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1106–1112, 1997.
  * [23] Jean-Yves Bouguet.  Camera calibration toolbox for matlab (2008).  URL http://www. vision. caltech. edu/bouguetj/calib_doc, 1080, 2008\.
  * [24] Robbe, C., Nsiampa, N., Oukara, A., and Papy, A.  Quantification of the uncertainties of high-speed camera measurements.  Int. J. Metrol. Qual. Eng., 5(2):201, 2014.
  * [25] François Chollet et al.  Keras.  https://keras.io, 2015.
  * [26] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241, Cham, 2015. Springer International Publishing.
  * [27] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448–456, Lille, France, 07–09 Jul 2015\. PMLR.
  * [28] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In Proceedings of The 33rd International Conference on Machine Learning, volume 48, pages 1050–1059, 2016.
  * [29] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi.  A review of uncertainty quantification in deep learning: Techniques, applications and challenges.  Information Fusion, 76:243–297, 2021.
  * [30] Terrance DeVries and Graham W. Taylor.  Leveraging uncertainty estimates for predicting segmentation quality.  CoRR, 2018.
  * [31] Yarin Gal and Zoubin Ghahramani.  Bayesian convolutional neural networks with bernoulli approximate variational inference.  In Proceedings of The 6th International Conference on Learning Representations, 2016.
  * [32] Trevor Hastie, Robert Tibshirani, and Jerome Friedman.  The elements of statistical learning: data mining, inference, and prediction.  New York: Springer, 2009.
  * [33] C. E. Shannon.  A mathematical theory of communication.  SIGMOBILE Mob. Comput. Commun. Rev., 5(1):3–55, Jan 2001.
  * [34] A. Dutta and A. Zisserman.  The via annotation software for images, audio and video.  In Proceedings of the 27th ACM international conference on multimedia, 72:2276–2279, 2019.
  * [35] Le Zhang, Ryutaro Tanno, Mou-Cheng Xu, Chen Jin, Joseph Jacob, Olga Cicarrelli, Frederik Barkhof, and Daniel Alexander.  Disentangling human error from ground truth in segmentation of medical images.  In Advances in Neural Information Processing Systems, volume 33, pages 15750–15762. Curran Associates, Inc., 2020.
  * [36] Simon Kohl, Bernardino Romera-Paredes, Clemens Meyer, Jeffrey De Fauw, Joseph R. Ledsam, Klaus Maier-Hein, S. M. Ali Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger.  A probabilistic u-net for segmentation of ambiguous images.  In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [37] L. Deng.  The mnist database of handwritten digit images for machine learning research.  volume 29(6), pages 141–142. IEEE signal processing magazine, 2012.
  * [38] O. Commowick, M. Kain, R. Casey, R. Ameli, J. C. Ferré, T. Kerbrat, A.and Tourdias, F. Cervenansky, S. Camarasu-Pop, T. Glatard, S. Vukusic, G. Edan, C. Barillot, M. Dojat, and F. Cotton.  Multiple sclerosis lesions segmentation from multiple experts: The miccai 2016 challenge dataset.  Neuroimage, 244:118589, 2021.