# Learning to Learn Unlearned Feature for Brain Tumor Segmentation

Seungyub Han, Yeongmo Kim, Seokhyeon Ha, Jungwoo Lee, Seunghong Choi $\dagger$
Electrical and Computer Engineering, Department of Radiology College of
Medicine $\dagger$
Seoul National University
$\\{$syhan, ymkim<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Work performed while at Hodoo AI

###### Abstract

We propose a fine-tuning algorithm for brain tumor segmentation that needs
only a few data samples and helps networks not to forget the original tasks.
Our approach is based on active learning and meta-learning. One of the
difficulties in medical image segmentation is the lack of datasets with proper
annotations, because it requires doctors to tag reliable annotation and there
are many variants of a disease, such as glioma and brain metastasis, which are
the different types of brain tumor and have different structural features in
MR images. Therefore, it is impossible to produce the large-scale medical
image datasets for all types of diseases. In this paper, we show a transfer
learning method from high grade glioma to brain metastasis, and demonstrate
that the proposed algorithm achieves balanced parameters for both glioma and
brain metastasis domains within a few steps.

##  1 Introduction

The performance of semantic segmentation using deep neural networks has been
improved recently. These segmentation networks are applied to the medical
image analysis to help doctors to save time in diagnosis. The state-of-the-art
networks still require large amounts of training data points for pre-training
and fine-tuning. Gathering medical image datasets, however, is an expensive
and time-consuming so that there are fewer datasets than the datasets for
common objects [1, 3, 7]. In particular, in brain tumor segmentation, there is
a proper dataset called BRaTS for the High Grade Glioma (HGG) and the Low
Grade Glioma (LGG) [5], and no well-annotated dataset for brain metastasis.

HGG and brain metastasis have the different structural feature but the similar
contrast feature. In contrast enhanced MR image, the region of brain tumor is
highlighted because of the contrast media. In both cases, tumors have the
similar contrast. These brain tumors have different pathological properties,
so they have different structural characteristics. Therefore, a pre-trained
network using the HGG dataset can not generate perfect segmentation for brain
metastasis. In this paper, We learn the unlearned feature of brain metastasis
without forgetting the pre-trained feature in order to optimize the balanced
parameters between HGG and metastasis.

We first pre-trained the fully convolutional network (FCN) [4] with HGG in the
dataset [5]. The gradient descent based fine tuning, which we call naive tune,
uses many selected data points which have balanced instances per each class to
produce the optimal fine-tuning results [3, 7]. We propose two novel fine-
tuning methods, passive meta-tune and active meta-tune, to optimize the pre-
trained network. These two methods decide which training data points are first
learned and update the network with [2]. The orders of training dataset is
determined with two active learning based rules, the passive learning and the
active learning [6]. In this work, we produce the annotated brain metastasis
data samples with 30 patients. Similar to the BraTS dataset, for a patient,
there are 4 MR sequences, FLAIR, T1, T1 contrast-enhanced, T2, and 25 slices
per each sequence.

##  2 Proposed methodology

(a) passive meta-tune method

(b) active meta-tune method

Figure 1: Two active learning based meta-tune methods : divide the results of
segmentation task into the good result tasks (learned) and the bad result task
(unlearned). At the first meta-tune step, the pre-trained parameter
$\theta_{0}$ usually lies on the sub-optimal region of the pre-training
optimum $\theta_{g}^{*}$. To reach the balanced target parameter
$\theta_{t}^{*}$, sample the order of meta-tune inputs by each method. (a) is
passive meta-tune that trains with randomly ordered inputs. The optimal
parameters $\alpha_{1},\beta_{1}$ for randomly sampled tasks lie in each
circle. The update direction depends on the combination of
$\alpha_{i},\beta_{i}$. (b) is DSC based order inputs. The first input is the
farthest data points across the optimums $\theta_{g}^{*},\theta_{b}^{*}$. The
induced update direction flows smoothly into $\theta_{t}^{*}$.

We propose two active learning based meta-tune methods, each of which is based
on random sampling and a variant of uncertainty sampling [6]. We define the
meta-tune as a fine-tune method using MAML algorithm [2]. The meta-tune
generalize the unlearned training examples more quickly than the graident-
based fined tune method (naive tune). As shown in Figure (1), we train the
model with the learned features continuously as well as the unlearned features
not to forget the learned features.

Algorithm 1 Active learning based meta-tune algorithms

1: Set learning rate hyper parameters : $\alpha,\beta$, pre-trained parameters
: $\theta$

2: Divide inputs into inner-loop data and outer-loop data, Order inputs by
passive or active method

3: for the meta batch size of one patient do

4: Decompose tasks into good task $\mathcal{T}_{g}$, bad tasks
$\mathcal{T}_{b}$

5: for $\mathcal{T}_{i}\in$ $\mathcal{T}_{g}\cup\mathcal{T}_{b}$ do

6: Sample one data point $D=(\mathbf{x},\mathbf{y})$ by each sample method

7: Compute
$\theta_{i}^{\prime}=\theta-\alpha\nabla_{\theta}\mathcal{L}_{\mathcal{T}_{i}}(f_{\theta})$

8: Sample one data point
$D_{i}^{\prime}=(\mathbf{x}^{\prime},\mathbf{y}^{\prime})$ by each sample
method

9: end for

10: Meta update
$\theta=\theta-\beta\nabla_{\theta}\sum_{\mathcal{T}_{i}}\mathcal{L}_{\mathcal{T}_{i}}(f_{\theta_{i}^{\prime}})$
using $D_{i}$

11: end for

##  3 Experimental Results

We clinically-acquire multimodal MRI scans and produce all the ground truth
annotations by neuroradiologists. We use a VGG-16 based FCN network as the
pre-train network. Then, we test our algorithms. (yellow: edema, green:
necrosis, brown: ehnacing tumor: brown, red : high probability).

(a) Pre-trained result on BraTS dataset Method | Dice Score ($\pm$std)
---|---
baseline | 0.66 (on BRaTS)
naive | 0.33 $\pm$ 0.3413
passive | 0.41 $\pm$ 0.2752
active | 0.45 $\pm$ 0.2317
(a) DSC result for enhaning tumor

(a) naive tune: all small tumors is detected, skull is misclassified as edema

(b) passive meta-tune: the region of edema is overestimated. this region
invades the part of skull

(c) active meta-tune show the minimum false positive area

(d) naive tune: unlike the previous case, a few step updates have no
generalization effect

(e) passive meta-tune: the worst case, which cannot capture a single tumor

(f) active meta-tune: the balanced optimal result

Figure 3: Segmentation results on the training dataset (a)-(c). The result of
passive meta-tune oscillates (b). Segmentation result on the validation
dataset (d)-(f).

##  4 Conclusion

We proposed an active meta-tune method which learns unlearned feature without
forgetting the original task. We show that our method have a generalization
effect within the target domain segmentation (brain metastasis).We expect our
method can be extended to other medical lesion applications.

### Acknowledgments

This work is in part supported by SNU Eng-Med Collaboration Grant, Basic
Science Research Program (NRF-2017R1A2B2007102) through NRF funded by MSIP,
Technology Innovation Program (10051928) funded by MOTIE, Bio-Mimetic Robot
Research Center funded by DAPA (UD130070ID), INMAC, and BK21-plus.

## References

  * [1] M. "Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A." Zisserman.  "the PASCAL Visual Object Classes Challenge 2011 (VOC2011) Results".  "http://www.pascal-network.org/challenges/VOC/voc2011/workshop/index.html".
  * [2] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International Conference on Machine Learning, pages 1126–1135, 2017.
  * [3] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  CoRR, abs/1405.0312, 2014.
  * [4] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  * [5] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  IEEE transactions on medical imaging, 34(10):1993, 2015.
  * [6] Stephen Mussmann and Percy Liang.  On the relationship between data efficiency and error for uncertainty sampling.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3674–3682, Stockholmsmässan, Stockholm Sweden, 10–15 Jul 2018. PMLR.
  * [7] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.


# The velocity structure of Cygnus OB2

Becky Arnold,1,2 Simon P. Goodwin1 and Nick J. Wright3
1Department of Physics and Astronomy, University of Sheffield, Sheffield S3
7RH, UK
2Department of Astronomy, University of Massachusetts, Massachusetts, 01003,
USA
3Astrophysics Group, Keele University, Keele, Staffordshire, ST5 5BG, UK
<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

The kinematic structure of the Cygnus OB2 association is investigated. No
evidence of expansion or contraction is found at any scale within the region.
Stars that are within $\sim$ 0.5 parsecs of one another are found to have more
similar velocities than would be expected by random chance, and so it is
concluded that velocity substructure exists on these scales. At larger scales
velocity substructure is not found. We suggest that bound substructures exist
on scales of $\sim$ 0.5 parsecs, despite the region as a whole being unbound.
We further suggest that any velocity substructure that existed on scales $>$
0.5 parsecs has been erased. The results of this study are then compared to
those of other kinematic studies of Cygnus OB2.

###### keywords:

stars: kinematics and dynamics – open clusters and associations: general –
stars: formation

††pubyear: 2019††pagerange: The velocity structure of Cygnus OB2–References

##  1 Introduction

Star forming regions are the focus of a great deal of scientific interest, and
for good reason. They inform our understanding of how stars are born and how
their environments evolve. Their study is also vital for our comprehension of
the conditions that planets form in, and the type/number of planets which may
exist in the universe.

Spatial and dynamical structure are perhaps the most important aspects
defining a star forming region, but can be difficult to interpret. A number of
statistical methods have been developed to quantify different aspects of the
spatial structure of these regions (Allison et al., 2009; Cartwright, 2009;
Maschberger & Clarke, 2011; Buckner et al., 2019), but they don’t touch upon
its velocity structure. However the Velocity Structure Analysis Tool (VSAT)
(Arnold & Goodwin, 2019) does, and it is used in this paper to investigate the
velocity structure of Cygnus OB2, which has previously had relatively little
statistical kinematic analysis.

Cygnus OB2 lies at a distance of approximately 1400 parsecs (Hanson, 2003;
Rygl et al., 2012; Berlanas et al., 2019), and has an estimated stellar mass
of order 104 $M_{\odot}$ (Drew et al., 2008; Wright et al., 2010). Estimates
of the region’s age vary, for example Massey et al. (1995) find an age of 1-3
Myr and Wright et al. (2015) find an age up up to 7 Myr with star formation
peaking 4-5 Myr ago. Given a number of estimates it seems relatively certain
that the age of the region lies somewhere between 1 and 7 Myr.

This region is chosen because it has been extensively studied, (Massey &
Thompson, 1991; Knödlseder, 2000; Comerón et al., 2008; Kiminki et al., 2015;
Roquette et al., 2017; Berlanas et al., 2018; Berlanas et al., 2019) meaning
there is a large amount of observational data already available. There have
also been studies focused on its spatial and kinematic structure (Wright et
al., 2014, 2016; Winter et al., 2019). This is useful as it allows findings
relating to the kinematic structure of the region which are achieved using
different techniques to be compared to see if they are consistent.

The structure of this paper is as follows. In section 2 the methods used to
collect and analyse the data are outlined. In section 3 the results of the
analysis are reported and in section 4 they are discussed. Finally in section
5 the conclusions drawn from the results are summarised.

##  2 Methods

###  2.1 Data collection

For this work we use the X-ray selected sample of Cygnus OB2 members presented
by Wright & Drake (2009) for the central portion of the association. X-rays
provide a largely unbiased diagnostic of youth that is effective for
separating young association members from older field stars, and Wright et al.
(2010) made further efforts to identify and remove foreground contaminants
from the sample. Proper motions for these stars were derived by Wright et al.
(2016) as part of the DANCe (Dynamical Analysis of Nearby Clusters) project
(Bouy et al., 2013).

The sample includes many of the known high-mass O-type stars in Cygnus OB2
with masses up to 100 $M_{\odot}$, as well as low-mass stars down to 0.1
$M_{\odot}$. The sample is estimated to be mostly complete to $\sim$ 0.8
$M_{\odot}$ (Wright et al., 2014). In contrast only around 70 per cent of
these sources appear in the $Gaia$ survey results and approximately half of
those do not survive $Gaia$’s recommended astrometric quality cuts (Arenou et
al., 2018; Lindegren, 2018; Lindegren et al., 2018). This is because the
sources are mostly fainter than is optimal for detection by $Gaia$.

The dataset used in this paper can be seen in Fig. 1. In this figure each star
is shown as a dot with a line. The location of the dot indicates the location
of the star and the length and direction of the line indicates the star’s
velocity. In the top right of the figure a velocity vector of 10 mas yr-1 is
shown with the median velocity uncertainty of the dataset shown by a grey
cone. This is done to give a visual representation of the uncertainties on the
velocity data.

Visual inspection of the figure shows a small number of stars with velocities
far greater than most. To investigate this we conduct an exploratory analysis
of the data. For additional details relating to the origins of this dataset
see Wright et al. (2016).

Figure 1: A plot of the data collected from the Cygnus OB2 region. The survey
area of the X-ray observations of Wright & Drake (2009) is outlined by a grey
dashed line. Each star is represented by a dot with a line. The position of
the dot indicates the position of the star in RA ($x$-axis) and Dec
($y$-axis). The length and direction of the line coming from each dot
indicates that star’s velocity vector. Stars which are removed from the sample
(as discussed in section 2.1) are shown by grey dots and vectors. In the top
right hand corner is a 10 mas yr-1 (equal to 66 km s-1 at 1.4 kpc) vector with
a grey cone outlining the median velocity uncertainty for the dataset.

###  2.2 Exploratory analysis

Fig. 2 shows a scatter plot of the stellar velocities in the reference frame
of the region. As was the case for Fig. 1 it is visually apparent there are a
small number of outliers. Additionally Fig. 3 shows a histogram with the stars
binned by the number of standard deviations their speed is from the mean
stellar speed of the dataset, and from this it is clear there is a small
number of stars which have speeds many standard deviations in excess of the
mean. Note that in Fig. 3 the $y$-axis is logarithmic in order to make bins
with few entries more easily visible.

Figure 2: The RA and Dec stellar velocities plotted against one another.
Figure 3: A histogram of the number of standard deviations stellar speeds are
from the mean stellar speed in the Cygnus OB2 dataset. On the $x$-axis is
number of standard deviations away from the mean. On the $y$-axis is the
number of stars in each bin of the histogram. In order to ensure bins with a
small number entries remain clearly visible the $y$-axis is logarithmic.

The presence of extreme velocity outliers is also noted in Wright et al.
(2016) where this dataset was first presented. That work concludes that these
outliers are most likely due to:

  * •

background/foreground stars that have been mistaken for members of Cygnus OB2;

  * •

stars ejected from Cygnus OB2 by dynamical events such as the disruption of
binary systems;

  * •

stars now in the region of Cygnus OB2 but originating in nearby star
clusters/associations which dispersed.

Regardless of their origin it is clear there is a small number of the 873
stars with very different velocities to the rest of the sample. Further, it
seems unlikely that they are representative of the region’s underlying
velocity structure. Due to this, stars with speeds more than three standard
deviations from the mean stellar speed in the dataset (a total of eight stars)
are removed. This removal has very little impact on the results, and the
effects it does have are discussed later. The removed stars are shown in grey
in Fig. 1 and outside the grey inset in Fig. 2

In Fig. 2 velocities appear to be centrally concentrated but, particularly in
the inset, there is no further obvious velocity structure apparent. To further
investigate the velocity structure we plot stellar speeds against their
distances from the centre, see Fig. 4. The centre is defined here as the mean
position of the stars.

Figure 4: A scatterplot of stellar speeds plotted against their distance from
the centre of the region. The moving median for this data is shown by a black
line.

There appears to be a peak in this distribution of points towards the middle
of the plot but it is difficult to assess meaningfully by eye. To better
understand it the moving median is calculated and is shown by a black line on
Fig. 4. Upon close examination the moving median appears slightly elevated at
distances between $\sim$ 2 and 3 pc, indicating stars with high speeds are
preferentially located at moderate distances from the centre of the field of
view. That said, this effect is extremely slight and the function is largely
flat indicating that any relationship between speed and position is weak.

###  2.3 Data analysis

The VSAT method (Arnold & Goodwin, 2019) is applied to analyse the velocity
structure of the dataset in more detail. In brief the method is as follows:
for every possible pair of stars this method calculates the distance between
them ($\Delta r$) and their velocity difference ($\Delta v$). To clarify, here
a ‘pair of stars’ does not necessarily refer to a binary system, but just to
any two stars in the region. The pairs are then binned by $\Delta r$ and
within each bin the average $\Delta v$ is calculated. Finally these are
plotted against each other. The uncertainties on the velocities are propagated
when calculating each $\Delta v$, and these errors are used to weight the
average. The impact of stochasticity on these uncertainties is also
incorporated, see Arnold & Goodwin (2019) for full details of how these
calculations are performed.

The VSAT method is applied twice, each time using a different definition of
the velocity difference between two stars $\Delta v$. The first case is
referred to as the magnitude definition, $\Delta v_{\rm M}$, and it is defined
as the magnitude of the difference between the two star’s velocity vectors.
Therefore for stars $a$ and $b$ $\Delta v_{\rm M}$ is calculated as:

$\Delta v_{a\\!b\rm M}~{}=~{}\lvert\mathbf{v}_{a}-\mathbf{v}_{b}\rvert$ (1)

This definition is particularly useful as a raw measure of how
similar/different stellar velocity vectors are.

The second way $\Delta v$ is defined is as the time differential of $\Delta
r$, i.e. the rate at which the distance between the stars is changing. This is
referred to as the directional definition, $\Delta v_{\rm D}$. For stars $a$
and $b$ this is

$\Delta v_{a\\!b\rm
D}~{}=~{}\frac{(\mathbf{r}_{a}-\mathbf{r}_{b})\boldsymbol{\cdot}(\mathbf{v}_{a}-\mathbf{v}_{b})}{\Delta
r_{a\\!b}}.$ (2)

In this definition if the stars are moving away from each other $\Delta r$
increases so $\Delta v_{\rm D}$ is positive, and if the stars are moving
towards each other $\Delta r$ decreases and $\Delta v_{\rm D}$ is negative.
This definition is particularly useful for studying if regions or sub-regions
are undergoing expansion/collapse. It as also useful for studying the relative
motions of different substructures within a region. Both of these things are
helpful in discerning the dynamical state of a region and its history.

In order to facilitate the calculation of these parameters the data is
converted from RA, Dec, and proper motions into a Cartesian coordinate scheme.
The skycoord class in the Python package astropy is used to perform this
conversion.

##  3 Results

###  3.1 Magnitude definition: $\Delta v_{\rm M}$

The VSAT method is applied to the dataset using the magnitude definition
$\Delta v_{\rm M}$. The results are shown by the blue line in Fig. 5. On the
$x$-axis of this figure is the distance between stars in parsecs, $\Delta r$,
and on the $y$-axis is the average magnitude of the difference between stellar
velocity vectors, $\Delta v_{\rm M}$, in km s-1.

To demonstrate the degree of statistical noise the velocity vectors are
randomly swapped between stars to remove any velocity structure. The method is
re-run and the results recorded. This is done 1000 times. The area containing
the central 1$\sigma$ of results is shown by a shaded grey region in Fig. 5.
Features in the non-randomised results (blue line) on scales smaller than the
spread of these randomised results (grey area) are not significant as they are
within the fluctuations in measured velocity structure due to pure
stochasticity.

Figure 5: The velocity structure of Cygnus OB2 as determined by the VSAT
method (Arnold & Goodwin, 2019) (blue line). The area containing the central
1$\sigma$ of the results of 1000 randomised cases is shown in grey. The
$x$-axis shows the separation between stars in parsecs $\Delta r$, and on the
$y$-axis is the average velocity difference of stellar pairs as defined by the
magnitude definition of velocity difference, $\Delta v_{\rm M}$, in km s-1.

Inspection of Fig. 5 shows a significant dip at scales $<$ 0.5 parsecs111This
feature is not visible if stars that are sigma clipped in section 2.1 are
included. This is because, due to their locations, these stars feature in a
disproportionately large number of low $\Delta r$ pairs, and the $<$ 0.5
parsec scale is particularly badly impacted. The sigma clipped stars have very
different velocities to the rest of the dataset so their over representation
causes an artificial increase in $\Delta v_{\rm M}$. This explains their
impact on the $<$ 0.5 parsec feature.. This means that stars closer to each
other than this tend to have similar velocity vectors relative to the
velocities across the whole region studied.

Between $\sim$0.5 - 5.5 parsecs the measured velocity structure (blue) is
almost completely flat; there is no change in the magnitude of the difference
between stellar velocity vectors as a function of how far apart they are.
Further, the small fluctuations from flatness that are observed are almost
entirely within the 1$\sigma$ bounds of those resulting from the regions where
velocity structure has been removed by random shuffling (grey area). Therefore
we conclude that there does not appear to be any velocity structure at these
scales.

For $\Delta r>$ 5.5 parsecs $\Delta v_{\rm M}$ dips again. However, the
results at these large scales are noisy and have high uncertainties. This is
largely due to the fact that only stars on the outermost edges of the dataset
are far enough apart to have such high $\Delta r$. Because of the morphology
of the region which has lower stellar density towards its outskirts this means
that there are fewer stellar pairs to populate these bins at large
separations, and the results at $\Delta r\gtrsim 5.5$ parsecs are highly
dependent on the exact velocity vectors of a small number of stars. Therefore
although the measured ‘structure’ exceeds the bounds of the results with
randomised velocity structures we cannot confidently determine whether
velocity structure is or is not present at $\Delta r$ beyond 5.5 parsecs.

###  3.2 Directional definition: $\Delta v_{\rm D}$

The VSAT method is now applied using the directional definition of the
velocity difference, $\Delta v_{\rm D}$. Recall that in this definition the
more rapidly stars tend to move away from each other the more positive $\Delta
v_{\rm D}$ is, and the more rapidly stars tend to move towards each other the
more negative it is. The results are shown by the blue line in Fig. 6222These
results are virtually indistinguishable from the results if the eight stars
removed in section 2.1 are included. The only impact is minor changes to the
exact fluctuations of the lines in Fig. 6, but the same overall trend is
observed. This is as expected: if the clipped stars are truly not members of
Cygnus OB2 they should have more or less random directions, so no net impact
on the results of the directional definition $\Delta v_{\rm D}$.. As in Fig. 5
the $x$-axis shows distance between pairs of stars, $\Delta r$, in parsecs.
The directional velocity difference $\Delta v_{\rm D}$ is given on the
$y$-axis in km s-1. As in section 3.1 velocity vectors are then randomly
shuffled between stars to remove any velocity structure, and the VSAT method
is re-applied. This is done 1000 times. The 1$\sigma$ boundary of the results
is plotted in grey on Fig. 6. This is done to give an idea of the amplitude of
apparent velocity structures which in fact result from statistical noise.

Figure 6: The velocity structure of Cygnus OB2 as determined by the VSAT
method (Arnold & Goodwin, 2019) (blue line). The area containing the central
1$\sigma$ of the results of 1000 randomised cases is shown in grey. The
$x$-axis shows the separation between stars in parsecs $\Delta r$, and on the
$y$-axis is the average velocity difference of stellar pairs as defined by the
directional definition of velocity difference, $\Delta v_{\rm D}$, in km s-1.

The velocity structure of the region as measured using the directional
definition of velocity difference does not fluctuate a great deal as a
function of $\Delta r$. The blue line in Fig. 6 is largely flat, and the
results are consistent with $\Delta v_{\rm D}=0$ (no net expansion or
contraction) for $\Delta r<3.5$ parsecs. For $\Delta r$ between 3.5 and 6
parsecs $\Delta v_{\rm D}$ is only very slightly above 0, and stays almost
entirely within the bounds of the randomised cases with no velocity structure.

At $\Delta r>6$ parsecs the results show a positive correlation between
$\Delta r$ and $\Delta v_{\rm D}$. Only stars on opposite sides of the dataset
and close to its edges are far enough apart to populate these high $\Delta r$
bins. Additionally, as stated, positive $\Delta v_{\rm D}$ indicates a stellar
pair is moving apart. Therefore this result implies that stars towards the
fringes of the dataset are moving outwards, i.e that the region is expanding
from it edges. However, for reasons discussed in section 3.1 VSAT is not
reliable beyond $\Delta r=$ 5.5 parsecs for this dataset because of low number
statistics. Therefore an additional test is conducted to verify whether this
apparent expansion is real.

First all the stars that are present in $\Delta r>6$ parsec bins (we will call
these fringe stars) are identified, and how many times they appear in those
bins is recorded. Next the centre of the dataset is determined by taking the
average position of all stars. The direction of each fringe star’s velocity
relative to the centre of the dataset is then calculated, with 0∘ indicating
the star is moving radially outwards, 90∘ that it is moving tangentially, and
180∘ that is moving radially inwards. If fringe stars are moving
systematically outwards then we expect their mean angle to be $<$ 90$\pm
2.90^{\circ}$. Note 2.90∘ is the expected standard deviation of the mean of a
uniform distribution between 0 and 180∘ given the number of datapoints.

The mean angle of the fringe star’s velocity directions is calculated. This
mean is weighted by how many times each star appears in $\Delta r>6$ parsec
bins, as recorded earlier. The mean is 88.00∘, however this is within the
expected standard deviation of 2.90∘ so therefore is not significant. From
this we conclude there is no evidence of expansion or contraction at any scale
in Cygnus OB2.

We emphasise that these findings do not constitute a null result. The fact
that the data does not convincingly deviate from the range occupied by the
randomised cases in Fig. 6 demonstrates that if there is any velocity
structure in the dataset according to the $\Delta v_{\rm D}$ definition then
it is so small as to be indistinguishable from random noise. As such the
principle finding of this section is that if there is any velocity structure
present in Cygnus OB2 is is less significant than random fluctuations.

This raises the question of how significant would the region’s expansion have
to be in order to be distinguishable from random fluctuations. In order to
test this regions with increasingly significant expansion are simulated, and
VSAT is applied. The regions are simulated by first drawing random positions
from a uniform distribution and velocities from a Gaussian distribution with a
width of 1, both centred on zero. Note the artificial regions are generated
with the same number of stars as the Cygnus OB2 dataset. Random radial
velocity components are then added to each star. The magnitudes of these
velocities are also drawn from a Gaussian with a width of 1 and a mean of
$R_{\rm{sig}}$, which we define as the significance of the outwards expansion.
This can be thought of as the ratio of systematic expansion to randomness.
Note that if the number drawn for the radial magnitude is negative the radial
component added will point inwards rather than outwards.

$R_{\rm{sig}}$ is increased in increments of 0.01. For each $R_{\rm{sig}}$ 100
regions are generated, VSAT is applied, and their $\Delta v_{\rm D}(\Delta r)$
is recorded. Results at $\Delta r>$ 1 are excluded as they rely on the
‘corners’ of the uniform distribution, and so rely on a smaller number of
datapoints than results at smaller $\Delta r$ so are less reliable. This is
analogous to the discounting of results at $\Delta r>$ 5.5 parsecs in the
Cygnus OB2 dataset’s results.

The $\Delta v_{\rm D}(\Delta r)$s of the expanding regions are then compared
to simulated regions that are not expanding in order to determine if VSAT can
differentiate them; another 100 regions that do not have systematic expansion
are generated and VSAT is applied. For each $\Delta r$ the range of $\Delta
v_{\rm D}$s that contain the central 1$\sigma$ of results is recorded. These
limits encapsulate the expected 1$\sigma$ fluctuation in $\Delta v_{\rm
D}(\Delta r)$ due to stochasticity (analogous to the grey regions in Fig. 5
and Fig. 6). The mean fraction of $\Delta v_{\rm D}(\Delta r)$ datapoints
outside of these limits for the expanding regions is recorded. As expected
this fraction increases with $R_{\rm{sig}}$, and increases beyond 1$\sigma$
significance when $R_{\rm{sig}}$ is 0.07. Therefore VSAT can identify
expanding regions if the ratio of systematic expansion to random velocity
dispersion is $\gtrsim$ 0.07.

For reference a simulated region with $R_{\rm{sig}}=0.07$ is shown in Fig. 7.
By eye the expanding nature of this region can not be readily observed. This
again demonstrates the need for quantitative techniques to analyse velocity
substructure in star forming regions.

As will be discussed further in section 4.3 the velocity dispersion of Cygnus
OB2 is difficult to characterise but it is likely of order 10s of km s-1.
Wright et al. (2016) finds a velocity dispersion of 17.8 km s-1, assuming this
is correct VSAT should be able to distinguish expansion $\geqslant$ 1.2 km
s-1, therefore any systematic expansion in the region must be below this
threshold.

Figure 7: A simulated region with an $R_{\rm{sig}}$ of 0.07. Each star is
shown as an arrow where the length and direction of the arrow reflects the
star’s velocity, and its position is the position of the star.

##  4 Discussion

###  4.1 Discussion of results using $\Delta v_{\rm M}$

The results of our VSAT analysis of this part of Cygnus OB2 can be summarised
as follows:

  * •

there is velocity structure on scales $<0.5$ parsecs;

  * •

scales $>0.5$ parsecs and $<5.5$ parsecs are consistent with a random velocity
field;

  * •

reliable conclusions cannot be drawn at scales $>5.5$ parsecs.

In discussing these results it is also important to recall that the velocity
dispersion of the data is of order 10s of km s-1, the size of the field of
view is of order 10 parsecs, and the age of the region is of order a few Myrs
(Massey et al., 1995; Wright et al., 2015). As such is is likely the majority
of the stars in the dataset did not form in this field of view but moved here
from their birth sites of order 10s of parsecs away, erasing their original
velocity structure. In this context the consistency of the results with a
random velocity field is to be expected as the distances between the stars
relative to one another represents a very small fraction of the total
distances the stars have traveled from their origins.

Nevertheless there is velocity structure on scales $<0.5$ parsecs. This
structure must either be primordial or have developed over time. It is
extremely difficult to see how such structures could be constructed in a
complex, unbound system such as a star forming region, so it is reasonable to
assume these structures are primordial, i.e. this signature is due to probably
fairly low-$N$ groups333It is not clear to us if they deserve the title
‘clusters’ of stars that formed near each other and so with similar velocities
according to the hierarchical paradigm of star formation (Elmegreen, 2000). We
can also conclude these groups must be bound as in order for unbound stars
with similar velocities to remain within around 0.5 parsecs of each other for
at least 3 Myr (the approximate age of the region) the would have needed to
form with a velocity dispersion of $\lesssim$ 0.17 km s-1.

To test this the local (within 0.5 pc) velocity dispersion of stars that are
overrepresented in $<$ 0.5 pc bins are compared to the velocity dispersion of
those that are not and are found to be a factor of 2.09 smaller on average.
This provides further evidence of kinematic groups on 0.5 pc scales. However
the velocity dispersions of stars overrepresented in $<$ 0.5 pc bins are many
times in excess of 0.17 km s-1, indicating the groups must be bound in order
to have remained so closely spatially associated. That said, it is likely that
some stars have become unbound from the groups since they formed due to
dynamical interactions, and that these groups formed with somewhat higher $N$
and larger size. However due to the youth of this region it is unlikely that
the $N$ and size of these groups has changed a great deal since their
formation.

Observationally the hypothesis of primordial bound groups with sizes of order
0.5–1 parsecs is supported by the fact this scale is the typical size of star
clusters (see e.g. Portegies Zwart, McMillan & Gieles (2010)), the size of a
typical molecular clump (Beuther et al., 2007), and is seen locally e.g. in
Taurus (Gomez et al., 1993). A large-scale random velocity field with many
‘embedded’ $\sim 0.5$ parsec bound groups is also consistent with the high
degree of spatial structure found by Wright et al. (2014) (they measure the
Cartwright & Whitworth (2004) $Q$-parameter to be 0.4–0.5). It also fits the
finding of Griffiths, Goodwin & Caballero-Nieves (2018) that the number of
wide massive binaries in Cygnus OB2 suggests many (at least 30) different
sites of massive star formation across the entire association.

###  4.2 Discussion of results using $\Delta v_{\rm D}$

As discussed in section 3.2 the velocity structure as measured using the
directional definition of velocity difference is very flat. $\Delta v_{\rm D}$
is consistent with 0 for $\Delta r$ $<$ 3.5 parsecs, indicating the region is
neither expanding or contracting on these scales. Further, for $\Delta r$
between 3.5 and 6 parsecs fluctuations in $\Delta v_{\rm D}$ are within the
bounds of the cases where any systematic velocity structure is removed by
random shuffling of the velocity vectors between stars (grey area in Fig. 6).
As such we conclude that any systematic expansion or contraction occurring
within the region at scales $<$ 6 parsecs is less significant than random
noise. Further VSAT is able to distinguish systematic expansion/contraction
from random noise if the ratio of the systematic component to the velocity
dispersion of the region is $\gtrsim$ 0.07, so if any exists it must be very
small.

At larger $\Delta r$ there is a positive correlation between $\Delta v_{\rm
D}$ and $\Delta r$. This indicates that at these scales stars further away
from each other tend to be moving apart, i.e. the region may be expanding from
the edges. However the VSAT method is not reliable at such large $\Delta r$
because of small number statistics so an additional statistical test is
conducted and no expansion is found.

As a result we conclude from Fig. 6 that there is no evidence of systematic
expansion or contraction on any scale in Cygnus OB2, and if any exists it is
far less significant than random noise. This makes sense if, as we argue in
section 4.1, the stars in this dataset formed across the Cygnus star forming
complex and happen to be in this field of view at the time of observation. If
this is the case we would not expect the stars to move in systematic
directions relative to one another, and indeed directional structure is not
observed in Fig. 6.

###  4.3 Comparison of results to other kinematic studies of Cygnus OB2

As has been mentioned Cygnus OB2 has been the subject of a number of other
kinematic studies. Here we compare our results to the findings of Wright et
al. (2014), Wright et al. (2016), and Winter et al. (2019).

In section 3.1 we find evidence of kinematic substructure at scales
$\leqslant$ 0.5 pc. In agreement with this Wright et al. (2016) applies a
number of statistics which confirm the presence of kinematic substructure in
the region. However from a by-eye inspection of the data Wright et al. (2016)
asserts that kinematic substructure is present at a range of scales which is
not consistent with our results. From the results of section 3.1 we argue that
Cygnus OB2 formed with a hierarchical, substructured morphology. This is in
agreement with Winter et al. (2019) which uses $N$-body simulations and
observations of the properties of protoplanetary disks in Cygnus OB2 to
constrain the initial conditions of the region, and finds the region must have
formed with substructure to reproduce the observations. Specifically Winter,
Clarke & Rosotti (2019) finds Cygnus OB2 may originate from a superposition of
multiple clusters, and their Figure 8(a) looks to the eye to be moderately
similar to our Fig. 1. However, it might well lack the $0.5$ parsec structures
we find in the real data (as that scale was not imposed in their initial
conditions and it is difficult to see how it could arise dynamically). A
possible avenue for future work could be to analyse a variety of different
idealised initial conditions to see which best-fit the real data.

We do a simple analysis and find the velocity dispersion of the region to be
28.72 $\pm$ 0.62 km s-1 which is not consistent with the result found by
Wright et al. (2016) which is 17.8 $\pm$ 0.6 km s-1. This disagreement is
explained by the fact the datast used here covers a larger region than Wright
et al. (2016). This discrepancy notwithstanding in both cases the region is
found to be unbound. Additionally Winter et al. (2019) (which also looks at a
larger region than Wright et al. (2016)) estimated the initial velocity
dispersion of the region to be 50 km s-1.

We conclude that the velocity dispersion is relatively high and is likely of
order 10s of km s-1. Given this and the region’s age we conclude some ‘mixing’
should have had time to occur, erasing velocity substructure on scales that
are not bound. This fits uncomfortably with the assertion of Wright et al.
(2016) that the region is not dynamically evolved. We argue that given the
relatively low-density nature of Cygnus OB2 that we would not expect stars to
have had a very large number of dynamical interactions despite traveling
potentially large distances. Nevertheless we expect some degree of dynamical
evolution to have occurred.

In Wright et al. (2016) the total kinetic energy of stars moving radially
inwards and outwards from the region are compared and found to be nearly
identical (the ratio is 51 per cent to 49 per cent). From this it is concluded
that there is no net expansion or contraction in Cygnus OB2. This result is
called into question by Winter et al. (2019) which argues that this ratio does
not provide a good measure of whether a region is expanding or contracting.
However in this paper, using an entirely different method, we also find that
there is no evidence of expansion or contraction in Cygnus OB2. This result
suggests that the region did not evolve to its current large, diffuse
structure by expanding from a significantly denser one as would be expected by
some models (Lada et al., 1991) (Carpenter, 2000).

This suggestion is in agreement with the conclusions drawn in Wright et al.
(2014) from their analysis of Cygnus OB2’s spatial structure. Despite its
consistency with previous results this finding is still odd. The Cygnus OB2
region has a velocity dispersion of 10s of km s-1 and, given its estimated
mass of $\sim$ 104 $M_{\odot}$, it should be supervirial. Conventional logic
states it should be expanding rapidly, but it does not appear to be doing so.
However from its studies of protoplanetary disk properties Winter et al.
(2019) estimates that gas removal ceased just 0.5 Myr ago and the region was
not originally unbound. It is conceivable that there has been insufficient
time for the region to ‘feel the effects’ of this change. This could explain
why it is not noticeably expanding yet, but may begin to do so in the near
future. This would be consistent with the conclusion drawn in Wright et al.
(2016) that within $\sim$ 4 Myr Cygnus OB2 will expand to be over 100 pc in
size.

##  5 Conclusions

We analyse the positions and proper motions of stars in a region of Cygnus OB2
from Wright et al. (2014) using the VSAT method presented in Arnold & Goodwin
(2019).

Our main findings are:

  * •

Stars within 0.5 parsecs of each other have significantly similar velocities.

  * •

At all reliable scales larger than 0.5 parsecs velocities are consistent with
a random distribution.

  * •

We find no evidence of systematic expansion or collapse in this part of Cygnus
OB2. If any such systematic motion exists its significance is $\lesssim$ a
factor of 0.07 of that of the velocity dispersion of the region, and cannot be
distinguished from random noise by the methods employed in this paper.

This suggests that we are observing many primordial bound structures on scales
$<0.5$ parsecs (‘groups’ or small ‘clusters’). However, any initial velocity
structure on scales larger than 0.5 parsecs has been erased by stars having
moved 10s of parsecs in the few Myr since they formed. Within this region we
see no significant evidence for global expansion or contraction (but we note
this is only the central part of Cygnus OB2).

## Acknowledgements

NJW acknowledges an STFC Ernest Rutherford Fellowship (grant number
ST/M005569/1). BA acknowledges PhD funding from the University of Sheffield.
Many thanks also to Stuart Littlefair for useful discussions and insights, and
the referee for helpful comments and suggestions which improved this work.

## References

  * Allison et al. (2009) Allison R. J., Goodwin S. P., Parker R. J., Portegies Zwart S. F., de Grijs R., Kouwenhoven M. B. N., 2009, MNRAS, 395, 1449
  * Arenou et al. (2018) Arenou F., et al., 2018, A&A, 616, A17
  * Arnold & Goodwin (2019) Arnold B., Goodwin S. P., 2019, MNRAS, 483, 3894
  * Berlanas et al. (2018) Berlanas S. R., Herrero A., Comerón F., Simón-Díaz S., Cerviño M., Pasquali A., 2018, A&A, 620, A56
  * Berlanas et al. (2019) Berlanas S. R., Wright N. J., Herrero A., Drew J. E., Lennon D. J., 2019, MNRAS, 484, 1838
  * Beuther et al. (2007) Beuther H., Churchwell E. B., McKee C. F., Tan J. C., 2007, in Reipurth B., Jewitt D., Keil K., eds, Protostars and Planets V. p. 165 (arXiv:astro-ph/0602012)
  * Bouy et al. (2013) Bouy H., Bertin E., Moraux E., Cuillandre J. C., Bouvier J., Barrado D., Solano E., Bayo A., 2013, A&A, 554, A101
  * Buckner et al. (2019) Buckner A. S. M., et al., 2019, A&A, 622, A184
  * Carpenter (2000) Carpenter J. M., 2000, AJ, 120, 3139
  * Cartwright (2009) Cartwright A., 2009, MNRAS, 400, 1427
  * Cartwright & Whitworth (2004) Cartwright A., Whitworth A. P., 2004, MNRAS, 348, 589
  * Comerón et al. (2008) Comerón F., Pasquali A., Figueras F., Torra J., 2008, A&A, 486, 453
  * Drew et al. (2008) Drew J. E., Greimel R., Irwin M. J., Sale S. E., 2008, MNRAS, 386, 1761
  * Elmegreen (2000) Elmegreen B. G., 2000, ApJ, 530, 277
  * Gomez et al. (1993) Gomez M., Hartmann L., Kenyon S. J., Hewett R., 1993, AJ, 105, 1927
  * Griffiths et al. (2018) Griffiths D. W., Goodwin S. P., Caballero-Nieves S. M., 2018, MNRAS, 476, 2493
  * Hanson (2003) Hanson M. M., 2003, ApJ, 597, 957
  * Kiminki et al. (2015) Kiminki D. C., Kobulnicky H. A., Vargas Álvarez C. A., Alexander M. J., Lundquist M. J., 2015, ApJ, 811, 85
  * Knödlseder (2000) Knödlseder J., 2000, A&A, 360, 539
  * Lada et al. (1991) Lada E. A., Depoy D. L., Evans Neal J. I., Gatley I., 1991, ApJ, 371, 171
  * Lindegren (2018) Lindegren L., 2018, in Recio-Blanco A., de Laverny P., Brown A. G. A., Prusti T., eds, IAU Symposium Vol. 330, Astrometry and Astrophysics in the Gaia Sky. pp 41–48, doi:10.1017/S1743921317005919
  * Lindegren et al. (2018) Lindegren L., et al., 2018, A&A, 616, A2
  * Maschberger & Clarke (2011) Maschberger T., Clarke C. J., 2011, MNRAS, 416, 541
  * Massey & Thompson (1991) Massey P., Thompson A. B., 1991, ApJ, 101, 1408
  * Massey et al. (1995) Massey P., Johnson K. E., Degioia-Eastwood K., 1995, ApJ, 454, 151
  * Portegies Zwart et al. (2010) Portegies Zwart S. F., McMillan S. L. W., Gieles M., 2010, ARAA, 48, 431
  * Roquette et al. (2017) Roquette J., Bouvier J., Alencar S. H. P., Vaz L. P. R., Guarcello M. G., 2017, A&A, 603, A106
  * Rygl et al. (2012) Rygl K. L. J., et al., 2012, A&A, 539, A79
  * Winter et al. (2019) Winter A. J., Clarke C. J., Rosotti G. P., 2019, MNRAS, 485, 1489
  * Wright & Drake (2009) Wright N. J., Drake J. J., 2009, ApJS, 184, 84
  * Wright et al. (2010) Wright N. J., Drake J. J., Drew J. E., Vink J. S., 2010, ApJ, 713, 871
  * Wright et al. (2014) Wright N. J., Parker R. J., Goodwin S. P., Drake J. J., 2014, MNRAS, 438, 639
  * Wright et al. (2015) Wright N. J., Drew J. E., Mohr-Smith M., 2015, Monthly Notices of the Royal Astronomical Society, 449, 741
  * Wright et al. (2016) Wright N. J., Bouy H., Drew J. E., Sarro L. M., Bertin E., Cuillandre J.-C., Barrado D., 2016, MNRAS, 460, 2593


# One-class anomaly detection through color-to-thermal AI for building
envelope inspection

Polina Kurtser  Kailun Feng  Thomas Olofsson  Aitor De Andres
<EMAIL_ADDRESS>
###### Abstract

We present a label-free method for detecting anomalies during thermographic
inspection of building envelopes. It is based on the AI-driven prediction of
thermal distributions from color images. Effectively the method performs as a
one-class classifier of the thermal image regions with high mismatch between
the predicted and actual thermal distributions. The algorithm can learn to
identify certain features as normal or anomalous by selecting the target
sample used for training. We demonstrated this principle by training the
algorithm with data collected at different outdoors temperature, which lead to
the detection of thermal bridges. The method can be implemented to assist
human professionals during routine building inspections or combined with
mobile platforms for automating examination of large areas.

###### keywords:

Building inspection , Thermography , Color-to-thermal , GAN , Anomaly
detection

††journal: Applied energy

[DI]organization=Department of Diagnostics and Intervention, Radiation
Physics, Umeå University,addressline=5A, H62, Norrlands Universitetssjukhus,
city=Umeå, postcode=901 85, country=Sweden

[IHBI]organization=Intelligent Human-Buildings Interaction Lab, Department of
Applied Physics and Electronics, Umeå University,addressline=X, Håken
Gullessons väg 20, city=Umeå, postcode=901 87, country=Sweden

[AF]organization=Department of Applied Physics and Electronics, Umeå
University,addressline=X, Håken Gullessons väg 20, city=Umeå, postcode=901 87,
country=Sweden

[F]organization=Department of Physics, Umeå University,addressline=Linnaeus
väg 24, city=Umeå, postcode=901 87, country=Sweden

##  1 Introduction

With various sustainability targets quickly approaching, an important effort
is underway to improve the energy efficiency of buildings, the single largest
energy consumer worldwide [1, 2]. Instead of focusing on improving design
standards for new buildings, which constitute only about 1% of the total stock
per year [3], the attention has steadily shifted to selectively refurbishing
existing buildings and their most inefficient components [4]. However,
characterizing the energy performance of each component and locating anomalies
is not an easy task: most buildings were constructed before proper efficiency
standards were enforced and detailed energy assessments are often missing [5].
On top of that, energy inefficiencies may also arise in newer buildings due to
factors such as lack of maintenance, improper installation, and natural
degradation. In this context, energy authorities and retrofitting companies
need available, effective, and fast methods to evaluate buildings and identify
inefficient or faulty components.

Passive infrared thermography (PIRT) is a powerful tool routinely used in the
context of civil engineering and building inspection [6]. It relies on imaging
the thermal radiation emitted by a target scene onto a two-dimensional
infrared (IR) detector and can, under certain assumptions, retrieve the
spatially-resolved target temperature. It has been proven effective in
locating energy efficiency anomalies in buildings such as thermal bridges, air
leaks, structural faults, and mould [6, 7, 8], as well as useful to measure
the environmental sol-air temperature [9, 10]. However, PIRT images
interpretation requires significant domain knowledge and is prone to
measurement artifacts arising from a complex interplay of factors such as
diffuse and specular reflections, angle of emission, wind speed, sky
temperature, and changes in emissivity [11, 12, 13]. As a result, despite
recent algorithmic advances, the execution of building inspection with PIRT
remains labor-intensive and time-consuming, with skilled professionals
required to interpret the data and discard artifacts [14]. While traditional
algorithms are commonly employed to process and enhance thermographic images,
their success when it comes to automating their interpretation has so far been
limited, especially in situations of high variability [15]. However, the rapid
development of artificial intelligence (AI) during the past decade has
revolutionized machine vision, offering vast opportunities that the field of
thermography for building inspection has so far exploited to a very limited
degree [16, 17].

The existing literature devoted to the intersection between AI and
thermographic analysis of building envelopes is rather limited [18, 19].
Current state-of-the-art focuses primarily on supervised learning approaches -
methods that rely on the meticulous annotation of RGB and thermal images,
attempting to encompass the variability using the acquisition of large
datasets. Garrido et al.[20] presented a method for thermal bridges and water-
related problems detection by employing a deep learning model (Mask R-CNN). To
improve the network detection performance, they introduce a pre-processing
step to the contrast between anomalous and non-anomalous areas of the building
envelope. Yang et al. studied different multimodal RGB-thermal fusion
techniques [21]. They concluded that using a UNet architecture to perform
encoding of a common latent space of the RGB and thermal images is beneficial
for detection of thermal issues such as detaching and missing tiles in
building facades. Alexakis et al. [22] showed how these deep network models
can be utilized for early identification of rising dampness at historical
monuments. In addition, literature in laboratory conditions includes the work
of Royuela et al., [23] who developed a method for the assessment of air
infiltration using convolutional and fully connected neural network models.
Similarly, Fang et al.,[24] used supervised, labeled based deep learning
architectures for detection of several anomalies.

It is difficult to pin-point why the literature devoted to applying AI methods
to thermal building envelope inspection is limited, especially in comparison
to other automation applications heavily relying on computer vision, such as
autonomous navigation, robotic perception, remote and local sensing, and
precision medical imaging. The low amount of publicly available datasets
acquired in uncontrolled conditions, the requirement of extensive labeling, as
well as the knowledge gap between building experts and AI professionals have
been discussed in the literature as possible causes [18, 16, 19]. Furthermore,
to the best of the author’s knowledge machine learning and AI tools have not
yet been applied in commercial building inspection, which suggests the
mentioned proof of concept models do not generalize well to the variability of
realistic scenarios. This lack suggests that commonly applied, data-driven,
supervised learning methods should potentially be replaced with alternative
methods of anomaly detection, less driven by expert knowledge. Thus, a
knowledge gap exist regarding how new or existing AI methods can be applied to
the inspection of building envelopes.

To bridge that gap, we build on an interesting approach reviewed by Pang et
al. for the deep representation of normality [25]. Specifically, the authors
review several methods for learning feature representations of normality using
deep autoencoders (AE) and generative (GAN) architectures. This is a
relatively recent development of the self-representation theory – if one
builds a model trained to represent normal data it will fail to represent
abnormalities. A deep auto-encoder-based anomaly detection method uses an
encoder (set of densely connected layers with stepwise decreasing
dimensionality) to first learn to map data samples into a latent space (the
last layer of the encoder with the smallest dimension) and then reconstruct
them to the original images by a decoder (another set of dense layers stepwise
increasing in dimensionality). Hence, auto-encoder trained on normal samples
would reconstruct normal images well but would fail to reconstruct abnormal
images. The reconstruction error (an error map between the original and
reconstructed images) is used as a metric for anomaly detection. These methods
show promising performances on benchmark datasets [26], while maintaining a
main strength - eliminating the need to label anomalies.

In this paper, we present a development on the deep representation of
normality approach by applying it to RGB-thermal image pairs. The technique
aims to overcome the stated challenges, especially for the generalization
problem and applicability to building inspection in uncontrolled environments,
data acquisition, and data labeling. We demonstrate it specifically for the
location of thermal bridges in building facades, but it can similarly be
utilized for the location of other anomalies. The technique can be used as it
is for assisting human professionals during manual inspections, or in
conjunction with mobile platforms and other classification schemes for the
automated inspection of large urban areas.

The article proceeds by briefly introducing the necessary AI frameworks and
tools. Further, it introduces and formalizes the algorithm structure. After
that, it goes into the detail of image acquisition, preprocessing, and network
training. Finally, the results are presented and discussed.

##  2 Introduction to relevant AI frameworks

###  2.1 Generative-adversarial networks

Generative deep learning networks are networks capable of generating new
samples closely resembling the distribution of the input data. While a number
of generative paradigms exists in the literature, in this paper we focus on
Generative-adversarial networks (GAN) - generative models in which two
networks, the generator and the discriminator, compete to outperform each
other [27, 28, 29]. Given a training data set T, the goal of the generator
network is to learn how to model T and to generate new synthetic data S with
the same statistical distribution as T. Meanwhile, the goal of the
discriminator network is to identify if a test data sample belongs to either T
or to S. During the training process, both networks evolve in parallel as the
generator learns how to fool the discriminator, and the discriminator learns
how to not be fooled by the generator. Nowadays, GAN algorithms are commonly
used in applications requiring the generation of highly realistic data.

###  2.2 Image-to-image translation networks

Image-to-image translation or style networks build upon the generative aspects
of GAN to perform domain transfer by generating an image in one domain to an
image in another. The idea of training a generative architecture on a non-
identical input-output image pair (thus not auto encoding but rather image-to-
image translation), has been explored in a number of network architectures.
For example, the generator of CycleGAN [30] can be realized through a ResNet
or U-net architectures. Both consist of encoder and decoder blocks realized in
convolutional layers and skip connections. While the ResNet based generator
also introduces a multi-modal translation part between the encoding and
decoding consisting of residual blocks. Some generative architectures such as
pix2pix [31], learns a mapping from input images to output images while the
mentioned above CycleGan architecture, an extension of the pix2pix lifts the
need to have paired multi-modal data samples. Extending even further on this
idea recent Transformer-based GANs utilize a self-attention mechanism,
originally proposed for a machine translation task in the generator part,
where the encoder and decoder networks are built using transformers [29].

###  2.3 Color-to-thermal networks

Color-to-thermal networks are a specific application of image-to-image
networks whose purpose is to translate an input color image (typically 3
channels, RGB) to their corresponding thermal distribution (single channel).
Kniaz et al. successfully demonstrated this approach using a dedicated
network, thermalGAN, to predict the thermal distribution of different objects
[32]. Later, Mizginov tested and compared the performance of different GAN
architectures when used as color-to-thermal [33]. Both studies included
buildings as a possible generation category and, thus, hold great promise for
the study of building facades.

###  2.4 One-class classification

One-class classification (OCC) is an algorithm classification paradigm that
aims to identify test objects as either positive (a.k.a. target) or negative
(a.k.a. outlier) [34]. Unlike traditional multi-class classification
algorithms, which typically require a collection of statistically significant
data for each possible category, most OCC algorithms are trained using only
positive data, learning to model it and to recognize anything outside the
trend as an outlier. This paradigm is useful when the negative data is either
not available, not enough to be statistically significant, or not well
defined. In the context of the paper, we aim to perform the detection of heat
leaks in RGB-thermal pairs using one-class classification, i.e. our model
training is reliant on anomaly-free RGB-thermal pairs for the identification
of the underlying normal distribution. In that sense, the training is loosely
supervised where image pairs are required for the training of the color-to-
thermal network, and labeling of the single class of anomaly-free thermal
images is required. But annotation of anomaly instances, a major bottleneck in
the implementation of these algorithms in the thermal inspection domain, is
not required.

##  3 Algorithm structure

The proposed method is inspired by the intuition-based reasoning of human
inspectors, who find anomalies based on experience-acquired expectations.

The algorithm (flowchart in Figure 1) utilizes pairs of aligned RGB ($x_{c}$)
and thermal images ($x_{t}$) of size $MXN$ to perform the detection of heat
leak instances. At first stage a mapping function $\Phi$ is learned to
estimate a thermal image $\widehat{x_{t}}$ such as:

$\widehat{x_{t}}=\Phi(x_{c})$ (1)

The mapping function is learned using a color-to-thermal translation network
for the generation of the estimated thermal image. Then, an anomaly metric map
($E$) is calculated as a function of the difference between the expected
thermal realization and the thermal image as sensed by the thermal camera,

$E=F(x_{t}-\widehat{x_{t}}),$ (2)

where $F$ is an identity function in the presented application.

To identify anomaly regions $A$ and subsequently detection of anomalous
objects the anomaly metric $E$ is thresholded with an input tolerance $T$ to
perform segmentation of the image areas with deviating anomaly metric - i.e.,
unexpected thermal measures, which constitute as anomalies.

$A_{i,j}=\left\\{\begin{matrix}0&E_{i,j}\leqslant T\\\
1&E_{i,j}>T&\end{matrix}\right.\forall i\in[1..N],j\in[1..M]$ (3)

Figure 1: Algorithm workflow in black, with the corresponding human-driven
steps in gray. After training A color-to-thermal GAN trained on data with the
desired properties first produces the expected thermal distribution of a
building envelope image (RGB). Then, the algorithm compares the expected and
actual thermal distributions. Strong mismatches between both are labelled as
anomalies.

To estimate the mapping function $\Phi$ and generate an expected thermal
realization $\widehat{x_{t}}$ from a corresponding RGB image $x_{c}$, we train
a pix2pix deep learning architecture [31]. The pix2pix architecture utilizes
the supervised paired relations between the aligned RGB and thermal images and
has been tested in the literature for the color-to-thermal image mapping
problem, albeit with mixed results [33]. Nevertheless, it has also shown to be
highly successful in image-to-image translation for the generation of
realistic RGB facades from categorical labels using relatively small data sets
(606 images) [35].

It is important to highlight that finding a mapping function $\Phi$ by
training a pix2pix network on exclusively positive, anomaly-free data samples,
is expected to encompass the relations between RGB and thermal data in anomaly
free cases. Therefore, a careful selection of training data is critical for
shaping the algorithm to identify the anomalies of interest in a given
application. To illustrate this principle, we used data corresponding to three
different environmental conditions and demonstrated how this can shape the
model to identify thermal bridges in building facades.

##  4 Implementation

###  4.1 Data acquisition

Pairs of visual and thermal images were acquired using a visible (RGB) and a
thermal camera mounted together on the top of a ground vehicle (a car). The
visible camera was a thermally-stabilized, multi-lens, and aberration-
corrected stereo camera (Stereolabs, ZED2), with a field of view of 110o
horizontal x 70o vertical and a sensor resolution of 2x1920x1080 px. The
stereo functionality (depth-sensing) was chosen for future reference but not
used in this study. The thermal camera was a single-lens, uncooled, and
mechanically shuttered microbolometer (Seek thermal, Mosaic Core), with a
field of view of 56o horizontal x 42o vertical and a sensor resolution of
320x240 px. Due to its single lens configuration, the thermal camera suffered
significant imaging distortion, which was dealt with during the preprocessing
phase. The acquisition took place place by driving more or less randomly in an
undisclosed neighbourhood in Sweden, which was chosen for its relatively
varied architecture and construction period of their buildings. Several
acquisition rounds were performed at three different outdoors environmental
conditions and the data split in 3 corresponding sets, as detailed in table 1.
In each set, the available images were randomly assigned for either training
the neural network or for evaluation purposes. In all cases, the wind speed
was low ($<4m/s$) and there was diffuse (i.e., not direct) sun light exposure
during and for at least 2 hours before the acquisition. The acquisition of the
two winter data sets was performed during the day hours, with a partially
cloudy ski and high coverage of snow and ice on the street ground. The
acquisition of the summer set was performed at sunset time and with clear sky.

Table 1: Data sets and acquisition conditions Set name | Winter4 | Winter8 | Summer
---|---|---|---
Outdoors temperature (${}^{o}C$) | -4 | -8 | 17
Number training images | 948 | 865 | 184
Number evaluation images | 50 | 50 | 9

###  4.2 Preprocessing

After acquisition, the thermal and visual images were preprocessed in four
steps. First, the imaging distortion was removed from the thermal data using a
second order radial (barrel) model. The distortion parameters were first
guessed using using the method described in [36] with a heated chessboard
pattern and then manually fine-tuned to optimize their alignment (i.e., their
overlap with the RGB images). Second, the visual images were cropped to the
same field of view of the thermal camera and both were interpolated to a
spatial resolution of 512x512 px. Third, the thermal data (absolute
temperature, float precision) was encoded by making it relative to the
outdoors temperature in each case, blanking it outside the interval [-5, 10]
${}^{o}C$, rounding it to reduce its resolution to $0.5$ ${}^{o}C$, and
shifting it to the positive integer space. Notice that after this process,
each pixel in the thermal images can only take integer values between 0 and
30. Fourth and last, both visual and encoded thermal data were saved as 8-bit
(png) images.

###  4.3 Network training

Supervised training of the network model was performed using the pix2pix
Matlab implementation by [37]. We trained three neural models, one for each
data set, using the parameters detailed in table 2, with the rest being the
standard of the implementation. Data augmentation was implemented by
horizontal mirroring. While the networks Winter4Net and Winter8Net were
trained from the scratch using the respective data sets, the same approach
could not be used for the Summer network due to the low amount of available
images. In this case, we used the trained network Winter4Net as a starting
point for the SummerNet, performing only a rather soft retraining with the
Summer data set.

Table 2: Training parameters Network name | Winter4Net | Winter8Net | SummerNet
---|---|---|---
Generator learning rate | $10^{-3}$ | $10^{-3}$ | $2\cdot 10^{-4}$
Discriminator learning rate | $10^{-3}$ | $10^{-3}$ | $2\cdot 10^{-4}$
Number epochs | 150 | 150 | 30
Mini-batch size | 3 | 3 | 3

##  5 Results and discussion

We started by analyzing the performance of the network model to learn the
correlation between RGB images and their thermal distributions for given
environmental conditions. Figure 2 shows the comparison between the actual and
predicted thermal images produced by the networks Winter4Net (2(a)),
Winter8Net (2(c)), and SummerNet (2(e)) when evaluating the RGB images of
their corresponding test sets. Notice that the test data was not included in
the training in any case. The accuracy in the prediction performs well in many
cases and exceeds the results presented by [33]. We speculate that this
improvement could be due to a more optimal preprocessing applied in our case
before training, which we observed had great influence in the network
performance. More specifically, the higher spatial resolution, narrow dynamic
range, and reduced temperature precision were critical to achieve good
results.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: Performance evaluation of the trained networks (a,b) winter4Net,
(c,d) winter8Net, and (e,f) SummerNet when making predictions on their own
test data sets. The histogram plots show the average pixel deviation for all
the images in the corresponding test sets, with the red vertical lines
indicating the deviation for the displayed images. Scale bars in oC over the
outdoors temperature.

Qualitatively, the model seems to identify the windows of older looking
buildings as warm during the winter, due to a heat-flux from the indoors to
the outdoors, but cold during the summer, when there is little to no indoor-
outdoor heat-flux. It also learned to identify most basements as mildly warm;
Swedish residential buildings tend to have common laundry equipment in the
basement, which produces significant amount of heat all year round.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3: Anomaly detection when using the (a,b) Winter4Net on Winter8 data,
(c,d) SummerNet on winter4 data, and (e-f) SummerNet on winter4 data. The
histogram plots show the average pixel deviation for all the images in the
corresponding test sets, with the red vertical lines indicating the deviation
for the displayed images. Scale bars in oC over the outdoors temperature.

To quantify the deviation between the predicted and actual thermal
distributions, the mean of the absolute difference across the pixels was
calculated for all the images in the test sets. The results are plotted as
histograms in figures 2(b), 2(d), and 2(f) with red vertical lines indicating
the deviation of the displayed images for each case. The distribution peaks
below $0.5$ degrees for the networks Winter4Net and Winter8Net, i.e., below
the temperature resolution at a single pixel, but it seems to decay as a
negative exponential, which might indicate potential poor predictions at its
tail. Comparatively, the SummerNet shows a less accurate and reliable
performance, although this was expected due to the significantly less
available training data.

Next, we study how the network model performs when the environmental
conditions for training and evaluation are different. This provides us with
both insights into the generalization capabilities of the networks as well as
the ability to detect anomalies that manifest in certain environmental
conditions based on a network trained in alternative conditions. Figure 3
shows the comparison between predicted and actual thermal distributions, as
well as the thresholded difference between the two. Since we want the
algorithm to recognize thermal bridges, $F$ in equation 2 is taken as an
identity, and in all shown cases a tolerance (T) of 1 ${}^{o}C$ to blank the
difference and establish the anomaly was used. In figure 3(a), the Winter4Net
was used to evaluate the Winter8 data set. In this case, the difference
between actual and predicted thermal images emphasized the windows of the
facade and the snow, making the algorithm label them as anomalies.
Quantitative assessments of windows based on thermal imaging is generally a
challenging task due to the thermal radiation transmitted from the interior of
the building, and thus the detected window anomaly here presented must be
understood as a qualitative feature.

In figures 3(c) and 3(e), the SummerNet is used to evaluate the Winter4 and
Winter8 data sets, respectively. In this case, the model expects the ground
and the walls to be hot, which discards them as anomaly as we defined it.
However, most windows appeared hotter than predicted and labelled as
anomalies. What is more, some basements presented areas with unusually high
temperature in the actual thermal images, hinting at insulation faults a
localized high indoors temperature, and were correctly labelled as anomalies.

##  6 Conclusion

We reported on a label-free method for the identification of anomalies in
building envelopes. The results indicate that the pix2pix network model
learned the correlation between color and thermal images and can make
predictions of higher quality than were so far reported. We observed that an
adequate data preprocessing before training was critical to obtain the desired
results. When the algorithm is trained and evaluated in different conditions,
it can identify relevant anomalies for the inspection of building envelopes.
Specifically, we demonstrated this principle by recording training data at
different outdoor temperatures, which thought the algorithm to detect thermal
bridges. However, using data of targeted buildings can teach the algorithm to
identify as normal the features of choice, such as those belonging to a
specific energy efficiency classification.

With the proposed method, the gap between the controlled environment and the
complexity of the outdoor scenario is minimized. Less labour intensive labeled
data acquisition process facilitates smoother implementation in industrial
applications. Furthermore the experimental protocol does not require capturing
the wide range of variability of anomalies in the training data. This further
makes the implementation of the suggested methods more realistic in industrial
scenarios and enables further automation of thermal inspection.

## Funding

This work is supported by the Swedish Energy Agency (Energimyndigheten, grant
No P2021-00202 and P2022-00141), and by the Swedish Research Council for
Sustainable Development (Formas, grant No. 2022-01475).

## Acknowledgments

We thank Umeå Kommun, the Innovation Office (Innovationskontoret vid Umeå
Universitet), and Uminova Innovation for their contributions and support.

## References

  * [1] Energy Efficiency 2022, Tech. rep., International Energy Agency (IEA), Paris (Dec. 2022).
  * [2] Annual European Union greenhouse gas inventory 1990–2021 and inventory report 2023, Tech. rep., European Commission, DG Climate Action European Environment Agency (Apr. 2023).
  * [3] Hypostat 2023, a review of europe’s mortgage and housing markets, Tech. rep., European Mortgage Federation (Sep. 2023).
  * [4] S. D’Oca, A. Ferrante, C. Ferrer, R. Pernetti, A. Gralka, R. Sebastian, P. op ‘t Veld, Technical, financial, and social barriers and challenges in deep building renovation: Integration of lessons learned from the h2020 cluster projects, Buildings 8 (12) (2018) 174.
  * [5] K. Feng, W. Lu, Y. Wang, Q. Man, Energy-efficient retrofitting under incomplete information: A data-driven approach and empirical study of sweden, Buildings 12 (8) (2022) 1244.
  * [6] A. Kylili, P. A. Fokaides, P. Christou, S. A. Kalogirou, Infrared thermography (irt) applications for building diagnostics: A review, Applied Energy 134 (2014) 531–549.
  * [7] M. Martin, A. Chong, F. Biljecki, C. Miller, Infrared thermography in the built environment: A multi-scale review, Renewable and sustainable energy reviews 165 (2022) 112540.
  * [8] A. Abdelhafiz, A. Balabel, M. Alwetaishi, A. Shamseldin, U. Issa, I. Sharaky, M. Al-Surf, M. Al-Harthi, An innovative approach to check buildings insulation efficiency using thermal cameras, Ain Shams Engineering Journal 13 (5) (2022) 101740.
  * [9] K. A. Ohlsson, R. Östin, T. Olofsson, Step-transient method for measurement of the heat transfer coefficient at surfaces exposed to simulated building outdoor environments using the sol-air thermometer, Journal of Building Physics 42 (3) (2018) 373–387.
  * [10] K. A. Ohlsson, R. Östin, T. Olofsson, Sol-air thermometer measurement of heat transfer coefficient at building outdoor surfaces, in: Cold Climate HVAC 2018: Sustainable Buildings in Cold Climates, Springer, 2019, pp. 329–338.
  * [11] W. Minkina, S. Dudzik, Infrared thermography: errors and uncertainties, John Wiley & Sons, 2009.
  * [12] B. Lehmann, K. G. Wakili, T. Frank, B. V. Collado, C. Tanner, Effects of individual climatic parameters on the infrared thermography of buildings, Applied Energy 110 (2013) 29–43.
  * [13] K. Ohlsson, T. Olofsson, Quantitative infrared thermography imaging of the density of heat flow rate through a building element surface, Applied Energy 134 (2014) 499–505.
  * [14] M. Fox, S. Goodhew, P. De Wilde, Building defect detection: External versus internal thermography, Building and Environment 105 (2016) 317–331.
  * [15] I. Garrido, S. Lagüela, R. Otero, P. Arias, Thermographic methodologies used in infrastructure inspection: A review—post-processing procedures, Applied Energy 266 (2020) 114857.
  * [16] M. H. Shariq, B. R. Hughes, Revolutionising building inspection techniques to meet large-scale energy demands: A review of the state-of-the-art, Renewable and Sustainable Energy Reviews 130 (2020) 109979.
  * [17] F. Panella, A. Pirinu, V. Dattoma, A brief review and advances of thermographic image-processing methods for irt inspection: a case of study on gfrp plate, Experimental Techniques 45 (2021) 429–443.
  * [18] A. Sabato, S. Dabetwar, N. N. Kulkarni, G. Fortino, Non-contact sensing techniques for ai-aided structural health monitoring: a systematic review, IEEE Sensors Journal (2023).
  * [19] D. Gertsvolf, U. Berardi, M. Horvat, Aerial infrared thermography and artificial intelligence-based fault detection and diagnosis methods for building energy systems: a review of the state-of-the-art, in: ZEMCH 2021 International Conference, UAE, 2021, pp. 530–540.
  * [20] I. Garrido, S. Lagüela, Q. Fang, P. Arias, Introduction of the combination of thermal fundamentals and deep learning for the automatic thermographic inspection of thermal bridges and water-related problems in infrastructures, Quantitative InfraRed Thermography Journal 20 (5) (2023) 231–255.
  * [21] X. Yang, R. Guo, H. Li, Comparison of multimodal rgb-thermal fusion techniques for exterior wall multi-defect detection, Journal of Infrastructure Intelligence and Resilience 2 (2) (2023) 100029.
  * [22] E. Alexakis, E. T. Delegou, P. Mavrepis, A. Rifios, D. Kyriazis, A. Moropoulou, A novel application of deep learning approach over irt images for the automated detection of rising damp on historical masonries, Case Studies in Construction Materials (2024) e02889.
  * [23] A. Royuela-del Val, M. Á. Padilla-Marcos, A. Meiss, P. Casaseca-de-la Higuera, J. Feijó-Muñoz, Air infiltration monitoring using thermography and neural networks, Energy and Buildings 191 (2019) 187–199.
  * [24] Q. Fang, C. Ibarra-Castanedo, I. Garrido, Y. Duan, X. Maldague, Automatic detection and identification of defects by deep learning algorithms from pulsed thermography data, Sensors 23 (9) (2023) 4444.
  * [25] G. Pang, C. Shen, L. Cao, A. V. D. Hengel, Deep learning for anomaly detection: A review, ACM computing surveys (CSUR) 54 (2) (2021) 1–38.
  * [26] X. Yan, H. Zhang, X. Xu, X. Hu, P.-A. Heng, Learning semantic context from normal samples for unsupervised anomaly detection, in: Proceedings of the AAAI conference on artificial intelligence, Vol. 35, 2021, pp. 3110–3118.
  * [27] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, Advances in neural information processing systems 27 (2014).
  * [28] J. Gui, Z. Sun, Y. Wen, D. Tao, J. Ye, A review on generative adversarial networks: Algorithms, theory, and applications, IEEE transactions on knowledge and data engineering 35 (4) (2021) 3313–3332.
  * [29] S. R. Dubey, S. K. Singh, Transformer-based generative adversarial networks in computer vision: A comprehensive survey, arXiv preprint arXiv:2302.08641 (2023).
  * [30] J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223–2232.
  * [31] P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, Image-to-image translation with conditional adversarial networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125–1134.
  * [32] V. V. Kniaz, V. A. Knyaz, J. Hladuvka, W. G. Kropatsch, V. Mizginov, Thermalgan: Multimodal color-to-thermal image translation for person re-identification in multispectral dataset, in: Proceedings of the European conference on computer vision (ECCV) workshops, 2018, pp. 0–0.
  * [33] V. Mizginov, V. Kniaz, N. Fomin, A method for synthesizing thermal images using gan multi-layered approach, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 44 (2021) 155–162.
  * [34] S. S. Khan, M. G. Madden, One-class classification: taxonomy of study and review of techniques, The Knowledge Engineering Review 29 (3) (2014) 345–374.  doi:10.1017/S026988891300043X.
  * [35] R. Tylecek, The cmp facade database, Tech. rep., Research Report CTU–CMP–2012–24, Czech Technical University (2012).
  * [36] Z. Zhang, A flexible new technique for camera calibration, IEEE Transactions on pattern analysis and machine intelligence 22 (11) (2000) 1330–1334.
  * [37] J. Pinkney, pix2pix - image to image translation using generative adversarial networks, https://github.com/matlab-deep-learning/pix2pix (2024).