N_{box}=\text{ Number of predicted boxes}$ $\displaystyle L_{cls}=\text{
Binary cross entropy loss (cf. Equation~{}\eqref{eq:Cls_Loss})}$
$\displaystyle L_{box}=\text{ $smooth_{L_{1}}$ loss as defined in
\cite[cite]{[\@@bibref{}{Girshick2015}{}{}]} (cf.
Equation~{}\eqref{eq:Box_Loss})}$ $\displaystyle L_{mask}=\text{ Pixel-wise
cross entropy loss}$ $\displaystyle L_{rpn\\_cls}=\text{ Binary cross entropy
loss of the regional proposal network -- }$ calculated as shown in Equation
(11) $\displaystyle L_{rpn\\_box}=\text{ $smooth_{L_{1}}$ loss of the regional
proposal network -- }$ calculated as shown in Equation (12) Equation 14:
Instance segmentation – multi-task loss for end-to-end training.

Our Model Training Decision. Due to the beneficial effects of end-to-end
training on object detectors, we train our Mask R-CNN architecture end-to-end.
Consequently, our multi-task loss function is the same as Equation (14).

##### Hyperparameters.

Object detection and, consequently, instance segmentation are highly-convex
optimization problems which include thousands of parameters [1]. In the
current body of work, there are numerous options of optimizers for neural
networks, for instance, the Adam optimizer or RMSProp. The Mask R-CNN
architecture employs a stochastic gradient descent (SGD) optimizer [1]. This
optimizer type uses information of previous gradients to achieve an
oscillation reduction when being close to the minimum [1]. Furthermore, it is
one of the standard optimizers in the machine learning domain [20]. Certainly
there are more advanced optimizers proposed by the current body of work. But
still, object detectors commonly employ plain stochastic gradient descent
optimizers “without putting much thought into it ” [1].
Our Model Training Decision. We follow this common direction of the object
detection resp. instance segmentation research field (cf. i.a. [133, 64, 54])
and thus also employ SGD. Another advantage of employing SGD as an optimizer
is that our results become comparable to existing work.

Of the thousands of parameters to be optimized for loss convergence, learning
rate and batch size are considered the most important for object detection.

Learning Rate. The learning rate is the factor which determines the amount by
how much the gradient is updated during optimization [20]. According to
Agarwal et al. [1], in their survey, there is no concrete procedure which will
lead to the optimal or at least good learning rate. The quality pf the
learning rate depends on different factors – i.a. the employed optimizer, he
model’s architecture, and the batch size. So, for now, there is no other
option than a trial-and-error procedure. Machine learners try different
learning rates, observe the respective loss curves, and take the learning rate
with the lowest loss. In case, they would like to use a learning rate from
literature, the training procedures need to be comparable.

A learning rate too small is likely to result in a long training that could
get stuck, whereas too large one is likely to result in learning sub-optimal
weights too fast or in an unstable training. A common counter measure for
these issues is learning rate annealing [86, 178]. This strategy describes the
adaption of the learning rate after a fixed number of epochs or after the
validation loss has not changed after a fixed number of epochs (plateau
annealing). After $120.000$ epochs, Mask R-CNN reduces the learning rate by
$10$ starting off with an initial rate of $0.02$.

The learning rate also depends on the transfer learning approach (cf. Section
Transfer Learning). If the network learns from scratch the learning rate is
higher due to the random initialization of the model parameters. If the model
is fine-tuning a pre-trained model, a smaller learning rate is advisable since
the parameters are already pre-trained with respect to the optimization
objective – at least to some degree [50]. Consequently machine learner
commonly use a learning rate of $0.001$ for fine-tuning (cf. i.a. [54]).
_Our Model Training Decision._ As we will also use fine-tuning888 for details
on the rationals please refer to the Sections Transfer Learning and MS COCO
Dataset – i.e., we will have a comparable training procedure – we use a
learning rate of $0.001$.

Batch Size. The batch size determines how many tensors are batched during
training. Batching tensors means training multiple data items together by
concatenating them. Larger batch sizes reduce the training time while extra
large batch sizes like $8192$ also have a beneficial effect on the performance
[1]. Such large batch sizes, however, are not feasible for instance
segmentation since instance segmentation works on the entire image and so on
large tensors [1]. Moreover, Mask R-CNN’s additional calculations such as the
anchors raise the amount of needed memory. So, it is likely to run in out-of-
memory errors for large batch sizes. This leads to the object detection and
instance segmentation community training with small batch sizes [1]. Small
means here a value range of $1$ to $16$ $3$-channel images. Mask R-CNN trains
two images per GPU and it uses eight GPUs in total which results in a mini
match size of $16$ [64].
_Our Model Training Decision._ To identify our mini batch size, we stepwise
increased it and set it to the maximum not leading to out-of-memory errors.
This procedure resulted in a batch size of $4$ images per GPU. As GPU we used
a NVIDIA GTX 1660. This results in a tensor size of $[4,800,800,6]$.

##### Transfer Learning.

All state of the art instance segmentation architectures commonly use transfer
learning, also called pre-training [78, 1, 133, 50, 64, 53]. Transfer learning
became popular since it reduces training time and increases loss convergence.
It builds upon the idea to use an existing model which is trained based on a
dataset similar to the own for the initialization of the parameters [1].
Instance segmentation models are typically pre-trained on the MS COCO dataset
(cf. i.a. [64, 99]). The COCO dataset was created specifically for object
segmentation. For details on the COCO dataset please refer to the following
Section – Section MS COCO Dataset.

Transfer learning can be categorized into:

  * •

no transfer – i.e., learning from scratch

  * •

transfer learning – i.e., fine-tuning the parameters of an already trained
network

  * •

no learning – i.e., feature extraction

Learning from scratch requires random initialization of the networks
parameters [64]. This is not circumventable in case the dataset to train the
network on and the pre-trained dataset differ too much [1]. Learning from
scratch might even be preferable when the dataset to be trained is large
enough [1]. However, usually a pre-trained model should be preferred in
contrast to random initialization [1].

Transfer learning or fine-tuning uses the parameter of a pre-trained network
as initialization – i.e., the start of the training. Typically, during
transfer learning, the network parameters are updated with a smaller learning
rate since the pre-training coarsely updates the parameter and transfer
learning updates them on a detailed level. It is possible to update all or
only a subset of the network’s weights. This depends on the learning
objective. For instance, R. Girshick [53] found for the Fast R-CNN
architecture that the first layer of the neural network is data and task
independent, however, the remainder of Fast R-CNN’s convolutional backbone
requires fine-tuning for the specific task’s objective. Similarity, Appalaraju
et al. [6] only had to fine-tune the last two convolutional layers for
learning the similarity of images. Faster R-CNN [133] and Mask R-CNN [64]
start their fine-tuning at the fourth convolutional layer. The necessity to
only fine-tune the last, or deeper, layers is explainable by: The deeper
layers hold high-level information. They learn this high-level information by
aggregating information from shallow layers. Fine-tuning only deeper layers
uses the same information as the pre-trained network from shallow layers,
however, creates different high-level features from this low-level
information. These different high-level features lead to an adaption of the
model to different objects and datasets.
_Domain-specific fine-tuning_ deals with the fine-tuning of a network on
domain data [54]. Ji et al. [78] fine-tuned Mask R-CNN on the SC-2016 dataset
which contains buildings in order to learn the prediction of changes in
buildings.

Feature extraction is also based on a pre-trained model. But, in contrast to
transfer learning, it keeps most parameters fixed. Throughout the training,
the gradients of these fixed parameters are not calculated and consequently
the parameters are not learned. Instead, the features of the pre-training are
used. This is beneficial for training scenarios with small datasets which are
comparable to the dataset used for pre-training. Additionally, feature
extraction can save training time.

Our Model Training Decision. We will use transfer learning since our directed
acyclic graphs and our task of difference detection has domain specific
properties – e.g., the influence of the influence factors – which cannot be
learned from other datasets. However, due to the shapes of our graph elements
the COCO dataset is suitable for pre-training – for details cf. Section MS
COCO Dataset.

##### MS COCO Dataset – Pre-Training Dataset.

The COCO dataset is one of the largest datasets for instance segmentation
[99]. Further, it is publicly available and machine learners commonly employ
it for pre-training. It contains $328.000$ $256x256$ images with $>2.500.000$
labeled instances of $80$ categories. Each category contains $>10.000$ labeled
instances. The images are daily life images which depict, for instance, cats,
dogs, or a soccer match.

Since the images of the COCO dataset are daily life images, their features
differ from our directed acyclic graph images. The factor of shape which is
the most dominant influence factor sets this into perspective. Our dataset
contains nodes and edges which we visualizes as circles (nodes) and lines with
arrowheads at the end (edges). These shapes also appear in the COCO dataset.
Circles, for instance, appear in images containing a ball and clear edges in
images with knifes. If we consider the shape of the entire directed acyclic
graph – nodes and edges combined – there are manifold shape options which,
e.g., resemble a house.
Our Model Training Decision. So, a dataset like the MS COCO dataset is a
suitable pre-training dataset for our objective of learning human detected
structural differences in directed acyclic graphs.

##### Training Dataset Size.

Training Size | Precision | Recall | F1 Score | AP
---|---|---|---|---
$200$ | $0.639$ | $0.418$ | $0.921$ | $0.575$
$1000$ | $0.765$ | $0.620$ | $0.894$ | $0.732$
$2000$ | $0.873$ | $0.677$ |  $0.93$1 | $0.784$
$3000$ | $0.899$ | $0.738$ | $0.937$ | $0.826$
Table 3: Experiment and analysis – varying training dataset sizes: small
training datasets already show relatively good performance. Increasing dataset
size improves the performance metrics. (Table taken from [18]) Figure 15: PR-
curves for varying training dataset sizes – a larger training dataset size
improves the PR-curve and thus the AP (average precision) by increasing both
precision and recall. (Figure taken from [18])

The decision of how many data items of the entire dataset – in our case
$10.000$ annotated directed acyclic graph pairs (cf. Section 1.3) – to use for
training the machine learning architecture is challenging like the decision
for hyperparameters like the learning rate. As for the learning rate, there is
no concrete procedure for determining the training dataset size. While the
current body of work from the early 90s found that the training objective
influences the sample size [120, 132], there is the rule-of-thumb amongst
image classification practitioners that $1.000$ data items are sufficient for
image classification objectives [167]. In case of the application of data
augmentation the overall training dataset size is reduced since the affine
transformation of data augmentation result in additional data items and thus
additional data diversity. The benchmark dataset ImageNet [90] is the source
of this rule-of-thumb. ImageNet contains daily life pictures and $1.000$
instances per class. These images depict multiple shapes, colors, sizes, or
occlusions and are thus more complex than our images. Our images only contain
white background pixels, black pixels for the depiction of the graph elements
– nodes and edges – and blue pixels, in case of the difference images, to
depict the differences. As our images are less complex, we expect that we will
achieve already relatively good results with less than $2.000$ images.

Experiment & Analysis. Table 3 shows our experiment results. With a training
dataset size of $200$ we already achieve an F1 score of $0.921$ and an AP of
$0.575$. An increased number of data items positively affects especially the
precision, recall, and AP (average precision) of our network. The AP for a
training dataset size of $2.000$ is higher due to both an increased precision
and an increased. The PR-curve in Figure 15 shows this since the green PR-
curve is higher than the blue and the yellow one.
_Our Model Training Decision._ As an increase to a training dataset size of
$3.000$ rather marginally improves our performance metrics (cf. Figure 15 –
red PR-curve), we choose a training dataset size of $2.000$.

##### Data Augmentation.

directed acyclic graph Type | Augmentation – yes/no? | Precision | Recall | F1 Score | AP
---|---|---|---|---|---
tree | data augmentation | $0.868$ | $0.657$ | $0.930$ | $0.770$
tree |  no data augmentation | $0.864$ | $0.760$ | $0.913$ | $0.830$
sparse | data augmentation | $0.846$ | $0.595$ | $0.959$ | $0.736$
sparse |  no data augmentation | $0.895$ | $0.727$ | $0.955$ | $0.825$
Table 4: Experiment and analysis – data augmentation: for our learning
objective data augmentation is not beneficial. It especially negatively
affects the network’s recall for both directed acyclic graph types – tree-like
and sparse. (Table taken from [18])

(a) Including data augmentation for tree-like directed acyclic graphs does
decrease the network’s recall and thus its AP. (b) Including data augmentation
for sparse directed acyclic graphs has even a stronger decreasing effect than
the effect for tree-like directed acyclic graphs. Refraining from data
augmentation for sparse directed acyclic graphs improves AP by $0.05$ and
precision by $0.13$.

Figure 16: PR-curves for the evaluation of (no) data augmentation for tree-
like and sparse data. (Figure taken from [18])

Data augmentation creates data diversity and enlarges the database which helps
to avoid overfitting and improves the network’s generalization abilities [1].
Data augmentation applies affine transformations to the images of the dataset.
Examples of such transformations are rotation, translation, or scaling. There
are many possibilities for affine transformations, however, not all are always
useful for every dataset [1]. Vertical flip of a traffic sign, for instance,
may change the semantic meaning to the one of a different, existing traffic
sign [186]. Consequently, domain knowledge is required to decide which
transformation are useful. For our directed acyclic graphs, only affine
transformations are valid which could also be produced by a hierarchical,
Sugiyama-like, top-rooted layout algorithm because this is the layout
algorithm type of our choice (cf. Section – Graph Drawings). Transformations
which such a layout algorithm could not produce would lead to a tampering of
the learned features. So, rotations and and vertical flips are excluded. A
flip in vertical direction, for instance, would result in a bottom-rooted
directed acyclic graph. Furthermore, we avoid augmentations like image
cropping that would affect the visualization of the data. This is in line with
the approach of Wöhler et al. [168] who deal with scatterplot visualizations.
Varying the brightness, hue and saturation setting had no impact. So, the
augmentation methods we applied are: horizontal flip, scaling, and
translation. We apply these transformations in a predefined range with random
parameters to avoid inducing an unintended pattern which the network may learn
and to increase the data variability (cf. [59] for details).
However, as, for instance, according to Kobayashi et al. [85], “in many cases,
data augmentation is effective; however, in some cases, it not be”, we
evaluate the benefits of data augmentation for our learning objective first
before we finally the decide on whether to use the above discussed strategy or
not.

Experiment & Analysis. We evaluated data augmentation for both of our directed
acyclic graph types – tree-like an sparse directed acyclic graphs. As Table 4
shows, data augmentation especially negatively affects the network’s recall
which consequently results in a lower AP. For tree-like directed acyclic
graphs, data augmentation decreases the AP by $0.06$ (cf. Figure 16 (a) – blue
curve). For sparse directed acyclic graphs, it decreases the AP by $0.089$
(cf. Figure 16 (b) – blue curve). This negative impact of data augmentation
for our learning objective is explainable as follows: The transformations,
supposably, especially the scaling and translations produce data items which
are too different from the not augmented data items in spite of us having
chosen the transformation types and ranges based on domain knowledge. The
difference of augmented and not augmented data items is likely to result from
the strict node placement and layer assignment rules of hierarchical,
Sugiyama-like layouts. And translating the directed acyclic graph influences
the node placement while scaling the directed acyclic graph influences node
placement and layer distance.
_Our Model Training Decision._ As a consequence, we refrain from employing
data augmentation.

##### Validation Dataset Size.

As for the training dataset size or for the setting of hyperparameters like
the learning rate, there is no concrete procedure for determining the
validation dataset size. As, for instance, the research of Nguyen et al.
[115], various researchers work on procedure guidelines or size ratio
recommendations. However, the influence of the dataset characteristics which
is used for the formulating the guidelines resp. recommendations is not to be
neglected. For our data and its characteristics we decided for a validation
dataset size of $1.000$.

####  1.5.3 Mask R-CNN – Architectural Details

Figure 17: Schema of Mask R-CNN’s architectural components. (Figure taken from
[48]) Figure 18: Detailed schematic representation of our Mask R-CNN
architecture for the prediction of human detected structural differences in
directed acyclic graphs. (Figure taken from [18])

Mask R-CNN is an extension of Faster R-CNN [133]. So, it not only predicts
bounding boxes and classification but also segmentation masks (cf. Figure 17 –
mask branch). Due to Mask R-CNN being an extension of Faster R-CNN knowledge
from object detection is also applicable to Mask R-CNN.

As Figure 17 shows, Mask R-CNN consists of four parts:

  1. 1

a neural network as the backbone

  2. 2

a regional proposal network (RPN)

  3. 3

a layer aligning the regions of interest (RoIAlign)

  4. 4

the head which subdivides into

    1. 4.1

the mask branch which calculates the segmentation masks

    2. 4.2

the fully connected layers which

       * •

predict the best fitting bounding boxes (box regression) and

       * •

do the classification

Figure 18 shows our architecture of Mask R-CNN which we used to reach the
objective of learning human detected differences for directed acyclic graphs.

##### Input Data

The input data size of our basis Mask R-CNN architecture is $[800,800,3]$.

Our Architectural Decision. We change the input data size to $[800,800,6]$.
This is our implementation of image stacking along the RBG-channel to enable
Mask R-CNN to process image pairs. Herewith, we follow the approach of Wu et
al. [169] who proposed an object-based change detection approach for the
popular children’s game “Spot the difference”. Our task is not that different
from this game. While this game aims to detect different object classes –
cats, dogs, every day objects etc. – we aim to detect differences with respect
to just two object classes – nodes and edges.
During training, since we use a batch size of $4$ per GPU, the input data size
is $[4,800,800,6]$. The model inference phase (testing) only uses the image
tensor with a size of $[800,800,6]$.

_Target Dictionary – Supervised Feedback._ The target dictionary is a look-up
table in form of a json file holding the supervised feedback for the training
phase. It consists of:

  1. 1.

the directed acyclic graphs (path to the GraphML files of the directed acyclic
graphs of the directed acyclic graph pair – base graph ($G_{1}$) and
alternative ($G_{2}$))

  2. 2.

the images $I_{1}$ and $I_{2}$ of the respective directed acyclic graphs
belonging to the respective directed acyclic graph pair – base graph ($G_{1}$)
and alternative ($G_{2}$) (path to the PNG images of the respective directed
acyclic graph pair)

  3. 3.

the difference image $I_{diff}$ which is difference graph $G_{diff}$ (path to
the PNG images $I_{diff}$)

  4. 4.

the segmentation masks (path to the binary mask images which we save as NPY
files)

  5. 5.

the bounding boxes (list of bounding boxes per directed acyclic graph pair)

  6. 6.

the labels (list of labels per bounding box – $1$: “contains difference”; $0$:
“no difference contained”)

For details on the calculation of the segmentation masks, bounding boxes and
bounding box labels of the supervised learning feedback, please refer to
Section Calculation of the Segmentation Masks, Bounding Boxes, and Bounding
Box Labels of the Supervised Learning Feedback of the supplementary material
of this Section in [59].

##### Convolutional Backbone.

The neural network, as the backbone, extracts image features and returns them
in feature maps (cf. Figure 17 – 1). Feature maps are vectors – also called
tensors – containing information about the images learned. These tensors are
of varying size since they contain either low- or high-level features. High-
level feature tensors are smaller because they are learned on deeper layers
and so more convolutions are already executed on the tensors. Due the neural
networks task of feature learning, the neural network choice is crucial for
the entire instance segmentations architecture [1].

Mask R-CNNs discriminative power is a measure of its ability to classify,
localize, and segment instance [185]. The objective of classification aims to
learn information on “difference” and “no difference” – i.e, high-level
semantic information [185]. While the task of localization is the
discrimination of position and scale changes segmentation has to differentiate
change regions from other image regions which are not of interest for
difference detection based in instance segmentation [185]. Due to the varying
tasks, classification. localization, and segmentation require features with
varying characteristics. Since an object should be still detectable when it is
scaled or rotated, classification requires invariant999invariant = features do
not change when geometric transformations such as scaling or rotation are
applied features [96]. Since the difference object’s position and their
segmentation depend on its rotation and scaling, localization and segmentation
require equivariant101010equivariant = $f:S\rightarrow T$ is equivariant with
respect to g: for every $s$ in $S$ when $g_{T}f(s)=f(g_{S}s)$ with $g_{T}$ =
transformation in feature space of the CNN, $g_{S}$ = transformation in image
space, $S$ = image space features, $T$ = feature space of the CNN [96]
features. Consequently, instance segmentation requires both invariant and
equivariant features. This feature combination, also called feature fusion
[185] is achievable with deep residual neural networks (ResNet) [65]. Such
ResNet architectures introduce skip connections and skipped layers. These skip
connections pass features from shallow layers to deeper layers without
changing them. Mask R-CNN combines ResNet with a feature pyramid network (FPN)
which leads to a significant increase in speed and performance [64].

Feature Pyramid Network (FPN). Feature pyramid networks are the neural network
based version of traditional feature pyramids. They exploit the implicitly
defined feature hierarchy of neural networks. The hierarchy or pyramid is
defined by the fact that shallow layers learn low-level features while deeper
layers learn higher-level features [170]. FPNs build the image pyramid via
returning the different scales [98]. FPNs have a top-down and a bottom-up path
[98]. In the feed-forward pass, the bottom-up path builds the implicit image
pyramid via the execution of several convolutions at different scales ob a
single-scaled input. This leads to multiple feature maps with varying scale.
The top-down path combines multiple feature maps with the introduction of
lateral connections. These merge feature maps by scaling up the smaller high-
level feature maps to the same size of the bigger low(er)-level feature maps.
FPNs became state of the art since they lead to a significant localization
improvement due to the presence of high-level semantic at all scales [185].
These multi-scale feature maps are beneficial for our learning objective since
our differences and their size range from nodes to large(r) connected
components.
_Our Architectural Decision._ Since our images have the same size as those of
Lin et al. [98], we also train end-to-end on a Mask R-CNN architecture pre-
trained on the MS COCO dataset, and we also have a detection objective, it is
fair to assume that Lin et al.’s settings for the FPN are also valid for our
scenario. The parameter settings are:

  * •

Upscaling of the feature maps by a factor of $2$

    * –

resulting feature map scales: ${F*1,F*2,F*4,F*8}$

  * •

Number of output channels: $256$

We then evaluated how well node and edge features are detectable with these
settings. As the features from the FPN are used by the regional proposal
network to predict the RoIs, we evaluate the quality of the feature extraction
as a result of the RoI predictions’ quality. So, please refer to the
Experiment & Analysis Paragraph of the Regional Proposal Network (RPN)
Section.

Experiment & Analysis – ResNet50 vs. ResNet101. For the ResNet part of the
convolutional backbone, we compared ResNet50 and ResNet101. Both are common
choice for the feature extraction task [1]. RestNet101 is twice as deep than
ResNet50. So, ResNet101 remarkably increases the number of parameters,
training time and memory consumption while it only increases the AP by
$0.0.2$.
_Our Architectural Decision._ Consequently, we chose ResNet50. Another
advantage, we gained with this decision is that ResNet50 is available pre-
trained on the COCO dataset – both only the convolutional backbone and the
complete Mask R-CNN. The end-to-end pre-trained Mask R-CNN includes a pre-
trained regional proposal network and a pre-trained head.

Pre-Training | Precision | Recall | F1 Score | AP
---|---|---|---|---
no | $0.003$ | $0.021$ | $0.005$ | $0.000$
yes | $0.160$ | $0.981$ | $0.276$ | $0.397$
Table 5: Experiment and analysis – pre-training: pre-training Mask R-CNN on
the MS COCO dataset increases recall significantly ($+0.961$) and precision by
$+0.157$. (Table taken from [18])

Figure 19: PR-curve – pre-training: A pre-trained Mask R-CNN clearly
outperforms a not pre-trained one. (Figure taken from [18])

Experiment & Analysis – (No) Pre-Training. Here, we evaluated our assumption
that the MS COCO dataset is suitable as a pre-training dataset. Figure 19 and
Table 5 clearly show that this assumption is confirmed. The PR-curve of a pre-
trained network clearly outperforms the one of a not pre-trained network. The
pre-trained Mask R-CNN achieves an AP of $0.397$ while the not pre-trained one
has an AP of $0.000$. Furthermore, pre-training improves Mask R-CNN’s
precision and recall – especially the recall with $+0.960$. Pre-training has
also a significant effect on the loss convergence. The pre-trained Mask
R-CNN’s loss converged much faster. This is an indication that the pre-trained
parameters are already good for our directed acyclic graph image pairs. To
ensure that we have no issue with overfitting, we tested the AP. The AP
increases significantly which can be seen in the values of Table 5 – i.e., we
do not overfit by using pre-training.
_Our Architectural Decision._ Since pre-training is clearly beneficial for the
network’s performance, we use the on the COCO dataset pre-trained ResNet50
backbone.

The implementation of our backbone can be found in mask_rcnn.py.

##### Regional Proposal Network (RPN).

Figure 20: Schematic representation of how the regional proposal network
generates the anchors – i.e., the bounding box proposals. (Figure taken from
[48])

Figure 21: Schematic representation of the intersection over union metric.
(Figure taken from [18]) $IoU=\frac{a_{t}\cap a_{a}}{a_{t}\cup a_{a}}$ (15)
Equation 15:  Intersection over union metric formula.

After the extraction of the features, instance segmentation consists of two
parts [1]:

  1. 1.

the proposal part
This network part learns to predict bounding box proposal for the objects to
be detected.

  2. 2.

the classification, regression, and segmentation part
The second part then learns to predict the objects’ classes, to choose the
best fitting bounding box compared to the target bounding box, and to segment
pixels.

Here, we elaborate on the proposal part – the regional proposal network (cf.
Figure 17 – 2). The second part – also called head – is discussed in one of
the following paragraphs.

The RPN is a small network which takes the extracted features as input and its
output are bounding box proposals – localization of the rectangular boxes and
classification. These depend on the actual image features. This means, if
there are, for instance, only small objects depicted in the images a correctly
functioning RPN would only predict small bounding boxes surrounding the images
objects. The RPN feeds the feature maps through a convolutional layer [133].
This result then is piped into two sibling fully connected layers. The first
one predicts the bounding box coordinates. The second one – the classification
layer – predicts the probability, also called the networks confidence, of an
object to be detected being inside the predicted bounding box proposal. The
bounding box predictions are given as
$d^{b}=[d_{x}^{b},d_{y}^{b},d_{w}^{b},d_{h}^{b}]$ with $d_{x},d_{y}$ being the
deviation of the box center and $d_{w},d_{h}$ being the box width and height.
The probabilities $p=(p_{0},p_{1}...p_{K})$ are predicted with a Softmax
function over $K$ classes.

The RPN works based on the sliding window approach. Furthermore, since the
objects to be detected can be differently sized and have different aspect
ratios, the RPN employ anchors. Anchors are potentially matching bounding
boxes which the RPN generates for varying scales and aspect ratios. The RPN
generates at each sliding window center position anchors for an in advance
defined range of aspect ratios and scales (cf. Figure 20). To give an example
– an anchor scale of $[32^{2},64^{2}]$ pixels and anchor aspect ratios of
$1:1$ and $2:1$ produce $4$ anchors per sliding window position – i.e,
$32x32$px, $64x64$px, $32x16$px, $64x32$px. In case an image has a size of
$800x800$px this leads to the generation of $2.560.000$ anchors per image.

Intersection over Union (IoU). In the training phase, the RPN evaluates for
all anchors how well they match the target bounding boxes $t^{b}$. To evaluate
this, the RPN calculates the overlap of all anchors and the target bounding
box for all image objects. The overlap is defined as the ratio of the
intersection of both the anchor and the target bounding box and the union area
of both (cf. Equation (15) and Figure 21). This metric is referred to as the
“Intersection over Union Metric”. It is also part of the performance
evaluation metrics (cf. Section Performance Evaluation Metrics).
Anchors with an overlap greater than an in advance defined IoU threshold are
classified as containing an object (class $1$). Mask R-CNN classifies anchors
with an overlap $\geq 0.7$ as containing an object [133]. Anchors with $<0.3$
overlap are classified as not containing an object (class $0$) [133]. Further,
Mask R-CNN ignores anchors which are neither classified as containing an
object nor classified as not containing an object [133]. Due to the generation
of multiple anchors per sliding window center position, the overall amount of
anchor classes are biased to negative – class $0$ – anchors. To solve this
bias, $256$ anchors are randomly sampled from the class $1$ and class $0$
anchors [133]. The sampling aims for a $1:1$ ratio of both anchor classes
[133]. The sampled $256$ anchors, or Regions of Interest (RoIs), are then the
input of Mask R-CNN’s head.

Mask R-CNN’s RPN generates anchors with 5 scales – $32^{2}$, $64^{2}$,
$128^{2}$, $256^{2}$, $512^{2}$ – and three aspect ratios – $1:1$, $1:2$,
$2:1$. Herewith, it follows the architectural decision of Faster R-CNN [64].
Also for the RPN, it is possible to train it separately or end-to-end [64].
End-to-end training results in high quality region proposals for object
detection learning objectives [133].

If the convolutional backbone contains an FPN, as in the case of Mask R-CNN,
the RPN gets a set of feature maps at multiple scales as input. The RPN adapts
its region proposals to those feature maps with the best match to the anchors’
scale and aspect ratio. Figure 18 visualizes this with the red boxes – anchors
– and the outgoing gray arrows referring to the feature map with the best
match. Since Mask R-CNN compares anchors to feature maps, the RPN and the head
can share their weights [1]. Sharing weights denotes the reuse of learned
weight parameters of one model architecture part for another one [64, 140]. In
Figure 18, the weight sharing is illustrated by two gray arrows going out of
ResNet50-FPN.

Non Maximum Suppression (NMS). Since RPN generates anchors of multiple sizes
at each sliding window position, the region proposals overlap significantly.
To reduce this redundancy, we apply non-maximum suppression (NMS) during
training and inference. NMS is a post-processing step in which duplicate
proposals are rejected by thresholds [1]. Only the proposals with the highest
confidence and above an certain IoU threshold are retained ($IoU>0.5$ = value
for Mask R-CNN). The use of NMS does not sacrifice detection accuracy [133],
but it significantly reduces the number of proposals and thus the training
resp. inference time. Fast inference time is important when models are applied
in tools, such as visual comparison tools. For example, a fast prediction time
of a model applied in a visual comparison tool is important to provide
immediate feedback to the user. Since we also generate many redundant anchors,
we also apply NMS.

Our Architectural Decision. Due to our decision to train end-to-end like Lin
et al. [98], we can also employ the settings of Lin et al. [98] for our RPN.
The settings are those:

  * •

anchor area size: ${32^{2},64^{2},128^{2},256^{2},512^{2}}$ px

  * •

anchor aspect ratios: ${1:2,1:1,2:1}$

  * •

IoU threshold for RoIs containing an image object (class $1$): $IoU>0.7$ – in
our case: a human detected structural difference

  * •

IoU threshold for RoIs not containing an image object (class $0$): $IoU<0.3$ –
in our case: background

Due to the generation of multiple anchors per sliding window center position,
we also encounter the bias of the anchors towards anchors of class $0$. As a
consequence, we employ the already above explained sampling solution. We also
sample $256$ anchors with a $1:1$ ratio of class $1$ and class $0$ anchors
[133, 98].

_Non Maxima Suppression (NMS)._ Faster R-CNN [133] uses an $IoU>0.7$ while
Mask-R-CNN [64] uses an $IoU>0.5$. Since a higher IoU threshold leads to fewer
boxes and faster training resp. inference time, we use an IoU threshold of
$IoU>0.7$. Furthermore, since the number of RoIs is too large, we set the
number of RoIs to randomly selected $2000$ RoIs before applying NMS and $1000$
RoIs after NMS. This is in line with the approach of [98].

(a)

(b)

Figure 22: Two examples of final predictions of our Mask R-CNN architecture –
segmentation masks and bounding boxes are detected precisely around the node
and edge shapes of the human detected directed acyclic graph changes. (Figure
taken from [18])

Experiment & Analysis – RPN. We conducted our evaluation with the above
mentioned settings. The focus of our evaluation was whether the network learns
precise bounding boxes and masks as the RPN provides the region proposals
which are the candidate boxes in which the nodes and/or edges resp. the
connected components are likely to be located in – i.e., the quality of the
final predictions allows to draw conclusions about the quality of the RPN’s
proposals. The final predictions, Figure 22 depicts two examples, show that
the segmentation masks are closely around the objects to be detected – nodes
and/or edges resp. connected components and also the final bounding box
predictions surround the entire objects to be detected. In conclusion, our
network learns precise node, edge, and connected component features.
_Confirmation of Our Architectural Decision._ Since our network is able to
precisely predict human detected structural difference, we keep the above
explained settings.

##### Head

In alignment with Figure 17, the head covers the RoIAlign layer, the fully
connected layers and the mask branch.

RoI Alignment Layer (RoIAlign, cf. Figure 17 – 3). The RoI alignment layer
brings the differently sized feature maps to fixed spatial extent [64]. Mask-R
CNN does this by the RoIAlign operation. The RoIAlign operation is a further
development of the RoIPool operation [53]. RoIPool is based on aggregated bins
which provoke issues for the localization and for pixel-accurate segmentation
masks [64]. So, RoIAlign replaces the aggregation inside the bins with
bilinear interpolation. The RoIAlign layer of Mask R-CNN brings the feature
maps to a spatial extent of $[7x7]$ for the fully connected layers and to a
spacial extent of $[14x14]$ for the mask branch.
_Our Architectural Decision._ For the RoI alignment layer, we keep the
settings of He et al. [64] as it achieves state of the art performance for
instance segmentation. We set the sampling size – the number of the grid
interpolation points – to $2$. Herewith, we follow the standard practice [31].

Mask Branch (cf. Figure 17 – 4.1). For the calculation of the segmentation
masks, Mask R-CNN pixel-wise applies a Sigmoid function. The Sigmoid function
assigns the value $1$ to all pixels which a pixels of the image object to be
detected and $0$ to all other pixels. According to Ch. M. Bishop [20], Sigmoid
functions are standard for binary classifications. Mask R-CNN has a separate
mask branch for predicting masks independent of the FC layer probabilities for
avoiding a mask competition amongst classes [64] preferring masks with high
probabilities over others. Masks either can be predicted as one mask per class
(class-specific) or one mask per RoI (class-agnostic) [64]. The segmentation
masks are predicted in the mask branch by a classification layer. Mask R-CNN
predicts class-specific segmentation masks111111class-specific masks are a
list of $k$ binary images with $k$ being the number of classes in $K$ – in our
case $k=2$. But performance-wise, class-agnostic121212class-agnostic masks are
a list of $n$ binary images with $n$ being the number of RoIs masks barely
differ from the class-specific ones. Since the mask branch does not convolute
the feature maps to short output vectors, it keeps the spacial layout of the
feature maps by exploiting the pixel correspondence of the feature maps and
the masks which i.a. leads to a higher mask accuracy [64]. The mask branch
assigns a value of $1$ to every pixel of the feature map with a Sigmoid result
of $>0.5$ [64].
_Our Architectural Decision._ Here, we also follow the settings of He et al.
[64] as it achieves state of the art performance for instance segmentation.

Fully Connected Layers (FC Layer, cf. Figure 17 – 4.2). The FC layers predict,
based on information coming from the RoIAlign layer, the bounding box
coordinates, a class label, and a probability – also called classification
score. The prediction of the bounding box coordinates is a regression problem.
Hereby, the box’s deviation of the box center – $x$, $y$ – is predicted and
also the box’ width $w$ and height $h$ [64]. The FC layer responsible for
classification predicts for each RoI whether it contains an image object to be
detected (class $1$) or not (class $0$). The classification score is a
discrete probability for each of the $k$ classes – $p=(p_{0},p_{1},...,p_{k})$
with $k$ being the number of classes in $K$ – in our case: $p=(p_{0},p_{1})$.
The classification score is the result of the application of a Softmax
function. The Softmax function is commonly used for the prediction of
classification scores (cf. i.a. [50, 20, 65, 81]). The classification score is
a quantification of the network’s confidence in the respective RoI containing
an image object to be detected. A score of $90\%$ expresses a high confidence.
_Our Architectural Decision._ After features extraction and region proposal,
the FC layers of the head combine information and predict for each proposed
RoI four outputs: label, probability (prediction confidence), bounding box and
segmentation mask. The label expresses the class – $0$: “background/no
change”, $1$: “change”. The discrete probability distribution assigns a
probability to each class prediction $p=(p_{0},p_{1})$. This prediction
expresses the network’s confidence that the RoI either contains a change
(class $1$) or not (class $0$). The regression layer outputs the parameterized
form of the predicted bounding boxes for the regions which contain detected
differences. The mask layer predicts the segmentation masks – i.e., all pixels
belonging to one difference. We follow our predecessor implementations Fast
R-CNN [53], Faster R-CNN [133], and Mask R-CNN [64] and use two non-linear
(ReLU-acitvated) fully connected layers – fc6 and fc7 layer – which combine
the information of the RoIAlign layer. The output of the FC layers is fed into
the predictor131313The term predictor denotes the mask branch, the box
regression and the classification part of the network. which is a network part
that flattens the FC layers and predicts graph differences.
In contrast to the rest of the network, the predictor is randomly initialized.
While parameters for region proposals and aligned RoIs benefit from pre-
training, the predictor should be specific to our dataset and hence, not be
pre-trained.
Similar to the region proposal network (RPN), the head evaluates RoI proposals
based on the IoU overlap with the target differences. We assign equal
probabilities to differences and background by setting the $IoU>0.5$. We
follow parameter settings of Girshick et al. [54] since all preceding
architectures [53, 64, 133] that base on the initial work of Girshick et al.
[54] kept the head parameters the same.
As positive RoIs – RoIs containing directed acyclic graph changes – are
extremely rare compared to negative RoIs – RoIs with no differences – we
ensure a minimum fraction of $0.25$ positive RoIs. Following Lin et al. [98],
we set the batch size of detections to $512$. The number of box detections
depends on the number of changes in the data. Since the number of changes in
our dataset is limited to $[1-8]$ additions per directed acyclic graph pair,
the maximum number of detections is limited to $8$ changes per image. However,
a human could highlight more differences than actually exist. Consequently,
the network could predict additional differences. Therefore we set the upper
limit of the number of detection, by including a generous buffer of $\approx
90$ items, to $100$ allowing the network to predict additional boxes if
required.
During inference, we control the number of predictions with a lower
probability bound threshold. We set this threshold to $p>0.92$ as we only want
to have the differences in which the network has a high confidence. To set the
threshold, we compared the AP which was maximized by $p>0.92$. Overlapping
boxes are reduced with NMS IoU threshold of $IoU>0.5$.

####  1.5.4 Further Evaluation Insights

In the following, we elaborate on further insights which we gained throughout
the evaluation.

##### Class Imbalance.

Class Weights | AP | Precision | Recall | F1 Score
---|---|---|---|---
$[0.5,0.5]$ | $0.744$ | $0.894$ | $0.781$ | $0.834$
$[0.2,0.8]$ | $0.805$ | $0.758$ | $0.860$ | $0.806$
$[0.1,0.9]$ | $0.857$ | $0.676$ | $0.918$ | $0.779$
Table 6: Class imbalance – evaluation of three class weights as an imbalance
countermeasure: class weights of $[0.1,0.9]$ achieved highest AP even though
precision and recall are least balanced which is indicated by a lower F1
score. (Table taken from [18])

Figure 23: PR-curve – class imbalance: evaluation of three class weights as an
imbalance countermeasure. The use of class weights increase the recall but
decreases the precision. Class weights of $[0.5,0.5]$ are equal to having no
class weights (cf. blue PR-curve). Class weights of $[0.1,0.9]$ lead to the
best performance (cf. green PR-curve). (Figure taken from [18])

Our dataset is imbalanced towards background areas. This is a problem which
commonly occurs for object detection [1]. More precisely, we consider all
image areas that do not cover a directed acyclic graph difference as
background – i.e, this applies to all areas which which cover common graph
elements. As our data items only have $1-8$ changes and changes consist of
small and narrow objects, only a small part of the image actually cover
differences. Consequently, RoIs covering background (negative RoIs) dominate
RoIs covering directed acyclic graph differences (positive RoIs). This issue
is called class imbalance [1]. In order to set a focus on differences, i.e.,
the positive RoIs, we balanced the class entropy loss $L_{c}ls$,
$L_{rpn\\_cls}$. Herewith, we follow the proposal of Agarwal et al. [1]. We do
not balance the binary entropy loss $L_{m}ask$, as masks are only considered
for positive RoIs. We tested class weights for $[0.1,0.9]$, $[0.2,0.8]$,
$[0.5,0.5]$ for the classes $0$ and $1$. Class weights of $[0.5,0.5]$
expresses having no class differences as classes are equally weighted. Class
weights of $[0.1,0.9]$ foster class $1$ to a maximal extent and still consider
negative RoIs – background – with $10\%$. We weight background RoIs with a
factor $0.1$ to not ignore them. Class weights of $[0.2,0.8]$ show the trend
between having no class weights to class weights of the maximal extent. The
values are determined by maximizing evaluation metrics. Class weights of
$[0.1,0.9]$ improved the network recall from $0.781$ to $0.918$ and thus the
AP – i.e., those weights lead to the best performance (cf. Figure 23 – green
PR-curve). As a consequence, we chose those weights. Since a higher class
weight for class $1$ – differences – corresponds to favoring differences over
background boxes, the number of positive boxes compared to the number of
negative boxes is increased. Differences count as detections. Consequently, a
higher class weight for differences increases the number of detections. As
discussed before, an increased number of detections increases the network’s
recall but decreases its precision. This results in a lower F1 score since
precision and recall are less balanced. Table 6 also reflects that.

##### Hyperparameter Optimization.

For the hyperparameter optimization we observed the training and validation
loss. Herewith, we evaluated the model’s learning speed and whether the
hyperparameters support learning.

Optimizer. We employ a stochastic gradient descent optimizer [94]. Herewith,
we are in line the current body of work of the object detection domain [1]. We
set the optimizer’s momentum to $0.9$ and its weight-decay to $0.0005$.
Herewith we follow the work of He et al. [64] who initially proposed the Mask
R-CNN architecture.

Learning Rate. As we employ transfer learning, we start with a learning rate
of $0.001$. This learning rate is typical for training pre-trained models [2,
43, 57, 169]. With this small learning rate we ensure that the knowledge of
the already trained parameters is conserved [133]. We reduce the learning rate
by a factor of $10$ if the validation loss does not improve anymore after a
certain amount of epochs. We decreased the learning rate by factor $10$ after
$10.000$ iterations as the validation loss stagnated. This setting was best
with respect to convergence and non-overfitting.
To be complete, we tested also higher learning rates. They lead to $NaN$
values which are an indication that the gradient is too large. As the gradient
gets multiplied with the learning rate during backpropagation, a reduction of
the learning rate also reduces the gradient. We Higher learning rates of 0.1
have been tested. They resulted in nan values which indicates a too large
gradient. As the learning rate multiplies the gradient during backpropagation,
reducing the learning rate reduces the gradient as well.

Loss Convergence, Epochs, Early Stopping. Our training loss converged after
$6.000$ iterations. This is yet another indication that transfer learning
accelerates loss convergence and that a small learning rate of $0.001$ is
appropriate in our context. Ji et al. [78] reported similar number. Their loss
also converged after $6.000$ for the Mask R-CNN architecture domain-
specifically fine-tuned on the SC-2016 dataset. Also De Luca et al. [35], who
trained on graph drawing images, reported similar loss convergence numbers.
Consequently, we see our convergence values as comparable with respect to pre-
training and architecture and as reasonable in the context of training on
graph drawing images using a pre-trained Mask R-CNN architecture. As
additional epochs increased precision, we did not stop after $6.000$
iterations – i.e. $12$ epochs – but trained for $40$ epochs with a mini-batch
size of $4$ image pairs which took about $15$ hours. Initially, we set our
epoch to $50$. As change detection publications which employed Mask R-CNN
reported convergence values of $5-40$ epochs and literature training neural
networks on graph drawing imaged reported convergence values of up to $20$
epochs, we wanted to have a buffer and therefore decided for $50$ epochs.
However, our validation loss started increasing again after $40$ epochs which
is the reason why we employed early stopping and stopped after $40$ epochs.

####  1.5.5 Domain-Specific Fine-Tuning and Final Architecture

Figure 24: Schematic representation of the domain-specific fine-tuning
approach we evaluated. (Figure based on original Figure from [18])

To adapt our Mask R-CNN architecture pre-trained on the COCO dataset to the
characteristics of our domain data – directed acyclic graphs visualized as
node-link diagrams with circular nodes and arrow-headed edges – we employ
domain-specific fine-tuning. To figure out the appropriate fine tuning, we
tested in total three different approaches – see Figure 24:

  * (A)

MS COCO model fine-tuned directly for the human-like detected difference data

  * (B)

MS COCO model fine-tuned for

    * 1)

the GT differences

    * 2)

the human-like detected difference data

  * (C)

MS COCO model fine-tuned for

    * 1)

the random(GT) differences

    * 2)

the human-like detected difference data

Our data to be fine-tuned initially consisted of two datasets:

  1. 1.

the GT dataset

  2. 2.

the human-like detected differences dataset

We introduced both in Section 1.3 – Dataset Creation.

directed acyclic graph Density | Architecture | AP | Precision | Recall | F1 Score
---|---|---|---|---|---
tree-like | (A) | $0.862$ | $0.689$ | $0.920$ | $0.788$
tree-like | (B) | $0.310$ | $0.595$ | $0.468$ | $0.524$
tree-like | (C) | $0.857$ | $0.670$ | $0.922$ | $0.776$
sparse | (A) | $0.844$ | $0.605$ | $0.959$ | $0.742$
sparse | (C) | $0.864$ | $0.663$ | $0.944$ | $0.779$
Table 7: Experiment and analysis – domain-specific fine-tuning: Domain-
specific fine-tuning the human-like detected differences with an advance fine-
tuning on the GT difference dataset (architecture (B)) clearly underperforms
compared to the other approaches. For tree-like directed acyclic graphs there
is only a marginal difference for directly fine-tuning on the human-like
detected differences (architecture (A)) or an in advance fine-tuning on the
random(GT) dataset (architecture (C)). However, for sparse directed acyclic
graphs, architecture (C) is recognizably better. (Table taken from [18])

(a) Architecture (B) recognizable underperforms compared
to architecture (A) and (C). There is only a marginal
difference in performance for training on tree-like directed acyclic graphs
when it comes to architecture (a) and (C).

(b) For sparse directed acyclic graphs, architecture (C) is recognizably
better than architecture (A).

Figure 25: PR-curve – domain-specific fine-tuning: Comparison of the different
domain-specific fine-tuning approach shown in Figure 24. (Figure taken from
[18])

Experiment & Analysis – Domain-Specific Fine-Tuning. Architecture (A) directly
fine-tunes on the human-like detected differences dataset. Herewith,
architecture (A) requires only one training. As especially real human detected
differences data may be limited, We came up with training approach (B). Here,
we operate under the idea that a two-stage fine-tuning – 1) fine-tuning on GT
dataset, 2) fine-tuning on human-like detected differences dataset – is
beneficial as features characterizing nodes and edges can be learned from the
larger GT differences dataset and then be used throughout the adaption to the
differences humans would detect. During training, we realized that
architecture (B) learns to detect all graph theoretic difference, however, is
not able to adapt to the differences humans would detect when the GT model is
fine-tuned based on the human-like detected differences. This is supported by
the PR-curve of architecture (B). In Figure 25 a), we can clearly see that the
orange PR-curve is way worse than those of architecture (A) and (C). As a
consequence, we introduced the random(GT) dataset and architecture (C).
Architecture (C) follows the same idea as architecture (B) but uses the
random(GT) dataset which is a random sampling of the GT differences which
finally makes the random(GT) more comparable to the human-like detected
differences dataset than the GT dataset (cf. Section 1.3 – Dataset Creation
for details).
For tree-like data, we can see in Figure 25 a) that there is only a marginal
difference in the performance of architecture (A), the blue PR-curve, and (C)
– the green PR-curve. Table 7 supports this insight. However, when it comes to
sparse directed acyclic graphs, architecture (C), the green PR-curve, is
recognizably better than architecture (A) – the blue PR-curve (cf. Figure 25
b)). This may result from the fact that the network’s ability to generalize
for more complex directed acyclic graphs – here: sparse directed acyclic
graphs – benefits from the increased diversity brought by the random(GT)
dataset. Agarwal et al. [1] also discuss this effect of diversity on the
ability to generalize in their survey. As we cover with our work both
densities, cf. Section 1.3 – Density for details, we choose architecture (C).

Dataset | AP | Precision | Recall | F1 Score
---|---|---|---|---
tree-like | $0.857$ | $0.670$ | $0.922$ | $0.776$
sparse | $0.844$ | $0.605$ | $0.959$ | $0.742$
tree-like & sparse | $0.862$ | $0.647$ | $0.940$ | $0.767$
Table 8: Experiment and analysis – combination of tree-like and sparse
directed acyclic graphs: trained separately and especially combined, our
network achieves a performance which is comparable to related change detection
work with Mask R-CNN. An example is the work of Ji et al. [78] who achieve an
AP of $0.858$ for the detection of changes of buildings. (Table taken from
[18])

(a) Architecture (C) achieves an
average precision of $0.857$ on tree-
like dataset with a precision
of $0.670$.

(b) Architecture (C) achieves an
average precision of $0.844$ on
sparse dataset with a precision
of $0.605$.

(c) Architecture (C) achieves an
average precision of $0.862$ trained
on tree and sparse data with a
precision of $0.647$.

Figure 26: PR-curves for the final training experiment and analysis – training
of the final architecture (C) for tree-like, sparse, and combined – tree-like
& sparse – directed acyclic graphs. (Figure taken from [18])

##### Final Training Experiment & Analysis – Combination of Tree-Like and
Sparse directed acyclic graphs.

As tree-like and sparse directed acyclic graphs may appear together in a real
world application, we conduct a final experiment on our final architecture and
compare the networks performance on tree-like and sparse directed acyclic
graphs separately and combined. The results for the model trained on the
combined dataset – tree-like and sparse directed acyclic graphs – shows that
our chosen hyperparameters and architectural decisions work for the data
characteristics we aimed to cover. As we can see in Table 8, our model
achieves for all datasets and $AP\approx 0.8$ – tree-like directed acyclic
graphs: $AP=0.857$; sparse directed acyclic graphs: $AP=0.844$; tree-like &
sparse directed acyclic graphs combined: $AP=0.862$. The PR-curves in Figure
26 support this. For the combined dataset it achieves the best results over
all evaluation metrics: $AP=0.857$, $precision=0.647$, $recall=0.940$,
$F1\text{ }Score=0.767$ (cf. Table 8).

Since our model has a high average recall of $0.940$ and recall is a
quantification of whether all differences humans would detect are detected by
the network, we can state that our model is able to predict human detected
differences of directed acyclic graph pairs. However, our average precision
indicates that there are still a certain amount of false positives (cf. Table
8). An in depth discussion follows in Section 1.6. The slight imbalance of our
network’s precision and recall can also be seen in the F1 score of $0.767$.
But still, our high recall indicates that the human detected differences are
well predicted by our network. With our average precisions – tree-like
directed acyclic graphs: $AP=0.857$; sparse directed acyclic graphs:
$AP=0.844$; tree-like & sparse directed acyclic graphs combined: $AP=0.862$ –
we achieve a performance which is comparable to the performance of related
change detection work with Mask R-CNN. An example is the work of Ji et al.
[78] who achieve an AP of $0.858$ for the detection of changes of buildings.

###  1.6 Discussion, Conclusion, and Future Work

Our discussion, conclusion and elaboration on future work are three-partite.
It covers the learning of human detected structural differences, image-based
learning, and capturing the human notion of structural differences.

####  1.6.1 Learning Human Detected Structural Differences in Directed Acyclic
Graphs

##### Result Summary.

As the task of change detection is similar to our learning objective and as we
decided to learn on picture due to humans also processing the visualizations
of the directed acyclic graphs, our basis architecture for learning the human
detected structural differences in directed acyclic graph pairs was a Mask
R-CNN pre-trained on the COCO dataset. One of our core architectural adaption
was to make Mask R-CNN work on image pairs. We achieved this by implementing
image stacking. Our final Mask R-CNN architecture is a domain-specific fine-
tuned network. Special to the fine-tuning is that before the network is fine-
tuned to the human-like detected differences dataset it is fine-tuned to the
random(GT) difference dataset. We introduced both datasets in Section 1.3. The
idea was that the random(GT) dataset increases the training dataset size as
especially human detected differences data may be limited. This idea was
beneficial – especially for the denser sparse directed acyclic graphs. The
final architecture has a constant and high prediction performance across
datasets – tree-like directed acyclic graphs: $AP=0.857$; sparse directed
acyclic graphs: $AP=0.844$; tree-like & sparse directed acyclic graphs
combined: $AP=0.862$.

##### Discussion.

With the AP of $0.862$ of our final architecture we can compete with related
work in the change detection domain which also employs Mask R-CNN and fine-
tunes it for domain-specific data. The work of Ji et al. [78] is an example.
The authors’ learning objective was the detection of building changes. They
achieve an AP of $0.858$. Ji et al. also trained with simulated data as we did
with our human-like detected difference segmentation masks. Our precision of
$0.647$, opposed to Ji et al.’s [78] ($0.833$) is worse by $0.186$. Our recall
is comparable – ours: $0.940$; Ji et al.’s: $0.922$. Our high recall is an
indication that nearly all human detected differences are predicted. The
decrease in precision may be due to the difference of our change detection
task and the one of Ji et al. [78]. While Ji et al. aimed to detect all
existing building changes we wanted to predict all human detected structural
differences which not necessarily have to be all existing structural
differences. This results from the influence factors having a hindering
effect. However, it might also be that the human detected differences
encompass all existing changes – i.e., the GT changes. This mostly happens
when there are no dense regions in the directed acyclic graphs. Because of
that, the network learns features and selection patterns which are more
dominantly representative for the human detected differences but also for the
GT differences and this may lead to the prediction of more changes than the
human detected ones leading to a decrease of the network’s precision. In
conclusion, this suggests that the final differentiating feature is still to
be detected.

##### Future Work.

As a high precision of is a quality measure for only predicting the human
detected differences, the decrease in our precision indicates that our network
predicts also additional differences. These additional differences are other
combinations of nodes and edges. Currently, there is a fixed IoU threshold
determining whether the prediction is a true positive or not. It might be a
worthwhile to investigate whether it is an option to use an adaptable IoU
threshold which reacts to where in the directed acyclic graph the prediction
is. This would mean that also the IoU threshold respects the human notion of
differences thus, for instance, in dense regions it is more likely that
smaller differences are spotted by humans as dense regions are hindering and
tend to mask edges while in sparser regions connected changes are more likely
to be seen by humans. However, in case the IoU is not adaptable and is set to
a relatively large number, e.g., to $>0.5$, small predictions are considered
as false positives in spite of them being reasonable due to the density of the
region. This, in turn, then negatively impacts the network’s precision. There
is no one right answer when it comes to the interpretation of commonalities,
differences, or similarities – neither from a graph theoretical nor a human
standpoint. And an adaptable IoU threshold, as we believe, could be a means to
consider that. Furthermore, it might be also worthwhile to further investigate
what differentiates human selection patterns and GT differences and under
which circumstances human selection patterns resemble the GT differences.
Knowledge on that would shed light on differentiating features which may have
the potential to avoid additional predictions decreasing precision.

####  1.6.2 Image-Based Learning

##### Result Summary

With Mask R-CNN we chose an image-based learning approach. While this is a
decision for not the most efficient approach with respect to training time, it
is a decision for the approach which is closest to how humans do their
comparison – i.e., humans work with the visualized directed acyclic graphs
when they visually compare them. The visualization resp. the visualization
design choices are known to influence human processing [180, 71].
Consequently, it is necessary to learn on the images.

##### Discussion

In case, one would like to have a certain independence of the visual design
choices, one could follow the approach of Wöhler et al. [168]. The authors
combined an image- and a structural-data-based network to learn the
correlation perception of humans. Consequently, they were were able to train
on both images and the structural data and achieved a certain independence of
the visualization. For pure image-based learning approaches, the learning
approach depends on the visual design choices. Consequently, it is not fair to
assume that the model would work for a varying visual design. This, however,
is currently no limitation of our work. In fact, at the moment, we refrained
from doing this since we identified our difference factors based on the
standard design – circular nodes and arrow-headed edges – for node-link
diagrams and have no information how the factors change for varying node-link
diagram designs.

##### Future Work.

However, it is an interesting future work question whether the integration of
a graph-based learning approach – e.g., a graph convolutional neural network –
would result in a more robust architecture. Maybe the integration of
additional structural graph information is even able to solve the precision
issue we discussed in Section Learning Human Detected Structural Differences
in Directed Acyclic Graphs. Pos-GCN [142] might be promising since it also
processes euclidean node positions. Since our graph data is laid out in
advance, we have the required euclidean embedding. This is a significant
difference to other graph-based learning approaches which mostly assume the
graphs not to be embedded in the euclidean space.

####  1.6.3 Capturing the Human Notion of Structural Differences

##### Result Summary and Discussion.

Our results for capturing the human notion of structural directed acyclic
graph differences rest on two legs:

  1. 1.

the data augmentation algorithm for the creation of a large enough training
dataset to train a robust and generalizable network

  2. 2.

the selection, adaption, and evaluation of a suitable learning approach – Mask
R-CNN

Data Augmentation. Our DFS-algorithm for data augmentation is a knowledge-
based algorithm which generates the supervised feedback – segmentation masks,
bounding boxes, and labels – for our learning approach. The augmented data
encodes human-like detected differences. By human-like we mean difference
annotations which are similar to those which an actual human would do. The
similarity is achieved by using a knowledge-based approach. Our DFS-algorithm
gains its knowledge from our studies on the visual comparison with respect to
differences (cf. [13, 14]). Our difference-coined comparison studies
identified six dominant factors which influence humans – some of them are
hindering while others foster the detection of differences. Our DFS-algorithm
uses these factors to decide which of the GT differences, or actually existing
differences, a human would detect. While the calculation of some of the
factors like edge crossing or visual symmetry is entirely clear and also
substantiated by the current body of work, the calculation of of other factors
like density shows room for improvement.

Our implementation of Mask R-CNN has shown to learn the patterns based on
which humans select resp. spot structural differences. Our recall indicates
that nearly all human detected differences are detected by our network, while
precision indicates that there are additional predictions which are not human
detected differences. For this we already discussed reasons and future work in
Section Learning Human Detected Structural Differences in Directed Acyclic
Graphs.

(a) The hindering factor
of density is the reason
that in the orange highlighted area only the nodes
are annotated as human-
like detected differences.

(b) Example 1 of an
area which would
rather obfuscate
edges.

(c) Example 2 of an area which would rather obfuscate edges.

Figure 27: Example which substantiates my future work idea to make the data
augmentation algorithm’s the decisions more sophisticated – especially those
based on the hindering factors. The behavior shown in Figure 27 a) is exactly
the correct behavior of my algorithm since this area is an area with a high
proportion of colored pixels. However, given the visual impression, this area
would rather not obfuscate any edges for a human. Figure 27 b) and c) show
examples of areas which would obfuscate edges for humans. (Figure based on
original Figures from [18])

##### Future Work.

For our data augmentation, an interesting future work step would be to make
especially the decision based on the hindering factors more sophisticated. To
explain this, we will use the example of density. In Figure 27, the hindering
factor of density is the reason that in the orange highlighted area only the
nodes are annotated as human-like detected differences. This is exactly the
correct behavior of our algorithm since this area is an area with a high
proportion of colored pixels. However, given the visual impression, this area
would rather not obfuscate any edges for a human. An area which would
obfuscate edges for humans would rather look like the orange highlighted area
in Figure 27. Consequently, it would be interesting and beneficial to
incorporate this details into data augmentation. To achieve this, there is
still a knowledge gap to close since, again exemplarily explained using
density, even the pixel density thresholds used for the decision on density
are an approximation inspired by research on scatterplots [17]. We had to use
this approximation, since, to the best of our knowledge, it is currently not
known in the current body of work what the precise human notion of pixel
density with respect to graph density is. When this knowledge gap is closed,
the parameters of our DFS-algorithm can be adapted to the research insights.
It is fair to draw inspiration from scatterplot research thus the nodes of
graphs and scatterplots are quite similar to each other.

Currently, Mask R-CNN learned the average selection patterns of humans for
structural differences in directed acyclic graph pairs. However, assuming that
each person shows certain differences in the differences they recognize, it
would be interesting to adapt our Mask R-CNN implementation to the individual
user in the future, in addition to using our learned model in an interactive
system. For this, the training phase of the network would have to be
reactivated and the supervised feedback would have to come from the user. The
user’s first response could be whether she agrees with the prediction. If yes,
the prediction is also taken as supervised feedback. If no, the user has to
mark the differences she detects and then these are taken as supervised
feedback for the network. Here, the challenges reside mostly in the
calculations of the supervised feedback based on the user’s feedback.

####  1.6.4 Conclusion

We propose an image- and segmentation-based change detection approach that is
able to predict and visually highlight structural differences – comparable as
to related visual comparison systems such as TreeJuxtaposer [114],
DifferenceMaps [7] or egoComp [100]. In contrast to those systems, we propose
an approach to incorporate the human notion by learning the human detected
differences. As we use a supervised neural network, the learning phase
requires the graph differences as labeled segmentation masks. After the
network is trained and then used only in the same context, e.g., for the same
user or for the same class of graph, the difference computation can be skipped
and the trained model can be used directly for prediction.

## Acknowledgements

We greatly benefited from the feedback of Prof. Ch. Kersting and Prof. Günther
Wallner. We would like to thank both of you very much for this.

## References

  * [1] S. Agarwal, J. O. D. Terrail, and F. Jurie.  Recent advances in object detection in the age of deep convolutional neural networks.  arXiv preprint arXiv:1809.03193, 2018.
  * [2] E. Ahmed, M. Jones, and T. K. Marks.  An improved deep learning architecture for person re-identification.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3908–3916. IEEE, 2015.
  * [3] L. Akoglu, H. Tong, and D. Koutra.  Graph based anomaly detection and description: A survey.  Data Mining and Knowledge Discovery, 29(3):626–688, 2015.  doi:10.1007/s10618-014-0365-y.
  * [4] B. E. Alper, B. Bach, N. Henry Riche, T. Isenberg, and J.-D. Fekete.  Weighted graph comparison techniques for brain connectivity analysis.  In Proceedings of the SIGCHI conference on human factors in computing systems, pages 483–492. ACM, 2013.
  * [5] K. Andrews, M. Wohlfahrt, and G. Wurzinger.  Visual graph comparison.  In Proceedings of the International Conference on Information Visualisation, pages 62–67. Institute of Electrical and Electronics Engineers Inc., 2009.  doi:10.1109/IV.2009.108.
  * [6] S. Appalaraju and V. Chaoji.  Image similarity using Deep CNN and Curriculum Learning.  arXiv preprint arXiv:1709.08761, pages 1–9, 2017.
  * [7] D. Archambault.  Structural differences between two graphs through hierarchies.  In Proceedings of Graphics Interface 2009, pages 87–94. Canadian Information Processing Society, 2009.
  * [8] D. Archambault and H. C. Purchase.  Mental map preservation helps user orientation in dynamic graphs.  In W. Didimo and M. Patrignani, editors, Graph Drawing, pages 475–486. Springer Berlin Heidelberg, 2013.
  * [9] D. Archambault, H. C. Purchase, and B. Pinaud.  Difference map readability for dynamic graphs.  In U. Brandes and S. Cornelsen, editors, Graph Drawing: 18th International Symposium, GD 2010, pages 50–61. Springer, 2011.  doi:10.1007/978-3-642-18469-7_5.
  * [10] A. N. Avanaki.  Exact global histogram specification optimized for structural similarity.  Optical Review, 16(6):613–621, 2009.  doi:10.1007/s10043-009-0119-z.
  * [11] B. Bach, E. Pietriga, and J.-D. Fekete.  GraphDiaries: Animated transitions and temporal navigation for dynamic networks.  IEEE Transactions on Visualization and Computer Graphics, 20(5):740–754, 2014.  doi:10.1109/Tvcg.2013.254.
  * [12] K. Ballweg, M. Pohl, G. Wallner, and T. von Landesberger.  A proposal for measuring the perceived pairwise similarity inspired by tversk’s similarity model on the example of directed acyclic graphs.  VIS Poster Track, 2017.
  * [13] K. Ballweg, M. Pohl, G. Wallner, and T. von Landesberger.  Visual similarity perception of directed acyclic graphs: A study on influencing factors.  In F. Frati and K.-L. Ma, editors, Graph Drawing and Network Visualization, pages 241–255. Springer International Publishing, 2018.
  * [14] K. Ballweg, M. Pohl, G. Wallner, and T. von Landesberger.  Visual similarity perception of directed acyclic graphs: A study on influencing factors and similarity judgment strategies.  Journal of Graph Algorithms and Applications, 22(3):519–553, 2018\.  doi:10.7155/jgaa.00467.
  * [15] F. Beck, M. Burch, S. Diehl, and D. Weiskopf.  A taxonomy and survey of dynamic graph visualization.  Computer Graphics Forum, 36(1):133–159, 2017.  arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.12791, doi:10.1111/cgf.12791.
  * [16] A. Bernstein, E. Kaufmann, C. Bürki, and M. Klein.  How similar is it? towards personalized similarity measures in ontologies.  In Wirtschaftsinformatik 2005, pages 1347–1366. 2005.
  * [17] E. Bertini and G. Santucci.  Is it darker? Improving density representation in 2D scatter plots through a user study.  In R. F. Erbacher, J. C. Roberts, M. T. Grohn, and K. Borner, editors, Visualization and Data Analysis 2005, volume 5669, pages 158–167. SPIE, 2005.  doi:10.1117/12.586930.
  * [18] A. Beyer.  Learning Perceived Structural Differences in Directed Acyclic Graphs (DAGs).  Master’s thesis, Technical University Darmstadt, 2020.
  * [19] S. Bhagat, G. Cormode, and S. Muthukrishnan.  Node Classification in Social Networks.  In Social Network Data Analytics, pages 115–148. Springer US, 2011\.  doi:10.1007/978-1-4419-8462-3{\\_}5.
  * [20] C. M. Bishop.  Pattern recognition and machine learning.  Springer, 2006.
  * [21] J. Blythe, C. McGrath, and D. Krackhardt.  The effect of graph layout on inference from social network data.  In International Symposium on Graph Drawing, pages 40–51. Springer, 1995.
  * [22] I. Borg and T. Staufenbiel.  Performance of snow flakes, suns, and factorial suns in the graphical representation of multivariate data.  Multivariate Behavioral Research, 27(1):43–55, 1992.
  * [23] R. Borgo, B. Lee, B. Bach, S. Fabrikant, R. Jianu, A. Kerren, S. G. Kobourov, F. McGee, L. Micallef, T. von Landesberger, K. Ballweg, S. Diehl, P. Simonetto, and M. Zhou.  Crowdsourcing for information visualization: Promises and pitfalls.  In D. Archambault, H. C. Purchase, and T. Hoßfeld, editors, Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments, pages 96–138. Springer International Publishing, 2017.
  * [24] N. G. Bourbakis.  Emulating human visual perception for measuring difference in images using an SPN graph approach.  IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics), 32(2):191–201, 2002.  doi:10.1109/3477.990875.
  * [25] S. Bremm, T. von Landesberger, M. Heß, T. Schreck, P. Weil, and K. Hamacher.  Interactive visual comparison of multiple trees.  In IEEE Symposium on Visual Analytics Science and Technology, 2011\. VAST 2011., pages 31–40. IEEE, 2011.  doi:10.1109/VAST.2011.6102439.
  * [26] A. Brunswig.  Das Vergleichen und die Relationserkenntnis.  BG Teubner, 1910.
  * [27] M. Burch, G. Andrienko, N. Andrienko, M. Höferlin, M. Raschke, and D. Weiskopf.  Visual task solution strategies in tree diagrams.  In 2013 IEEE Pacific Visualization Symposium, pages 169–176. IEEE, 2013.
  * [28] M. Burch, N. Konevtsova, J. Heinrich, M. Hoeferlin, and D. Weiskopf.  Evaluation of traditional, orthogonal, and radial tree diagrams by an eye tracking study.  IEEE Transactions on Visualization and Computer Graphics, 17(12):2440–2448, 2011.  doi:10.1109/TVCG.2011.193.
  * [29] J. A. Burkell and Z. W. Pylyshyn.  Searching through subsets: A test of the visual indexing hypothesis.  Spatial Vision, 11(2):225–258, 1997.
  * [30] R. Caye Daudt, B. Le Saux, and A. Boulch.  Fully convolutional siamese networks for change detection.  In Proceedings - International Conference on Image Processing, ICIP, pages 4063–4067. IEEE, 2018.  doi:10.1109/ICIP.2018.8451652.
  * [31] K. Chen, J. Pang, J. Wang, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Shi, and W. Ouyang.  Hybrid task cascade for instance segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pages 4974–4983. IEEE, 2019.
  * [32] Y. Chen, Z. Guan, R. Zhang, X. Du, and Y. Wang.  A survey on visualization approaches for exploring association relationships in graph data.  Journal of Visualization, 22(3):625–639, 2019.  doi:10.1007/s12650-019-00551-y.
  * [33] C. Collins and S. Carpendale.  Vislink: Revealing relationships amongst visualizations.  IEEE Transactions on Visualization and Computer Graphics, 13(6):1192–1199, 2007.  doi:10.1109/TVCG.2007.70521.
  * [34] D. W. Cunningham and C. Wallraven.  Experimental Design: From User Studies to Psychophysics.  AK Peters/CRC Press, 2011.
  * [35] F. De Luca, M. I. Hossain, and S. G. Kobourov.  Symmetry detection and classification in drawings of graphs.  In D. Archambault and C. D. Tóth, editors, Graph Drawing and Network Visualization: 27th International Symposium, GD 2019, pages 499–513. Springer International Publishing, 2019.
  * [36] C. D. Demiralp, M. S. Bernstein, and J. Heer.  Learning Perceptual Kernels for Visualization Design.  IEEE Transactions on Visualization and Computer Graphics, 20(12):1933–1942, 2014.  doi:10.1109/TVCG.2014.2346978.
  * [37] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
  * [38] S. Di Bartolomeo, A. Pandey, A. S. Leventidis, D. Saffo, U. H. Syeda, E. Carstensdottir, M. S. El-Nasr, M. A. Borkin, and C. Dunne.  Evaluating the effect of timeline shape on visualization task performance.  CHI ’20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020.
  * [39] E. Di Giacomo and A. Lubiw, editors.  Graph Drawing and Network Visualization, number 23. Springer, 2015\.  doi:10.1038/nrmicro.2016.131.
  * [40] S. Diehl and C. Görg.  Graphs, They Are Changing.  In International Symposium on Graph Drawing, pages 23–31. Springer, 2002.  doi:10.1007/3-540-36151-0{\\_}3.
  * [41] R. Diestel.  Graphentheorie.  Springer Verlag, 2017.
  * [42] E. Dimara, S. L. Franconeri, C. Plaisant, A. Bezerianos, and P. Dragicevic.  A Task-based Taxonomy of Cognitive Biases for Information Visualization.  IEEE Transactions on Visualization and Computer Graphics, pages 1413–1432, 2018.  doi:10.1109/TVCG.2018.2872577.
  * [43] A. Dosovitskiy and T. Brox.  Inverting Visual Representations with Convolutional Networks.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4829–4837. IEEE, 2016.
  * [44] V. Dumoulin and F. Visin.  A guide to convolution arithmetic for deep learning.  arXiv preprint arXiv:1603.07285, 2016.
  * [45] S. L. Franconeri.  The nature and status of visual resources.  The Oxford Handbook of Cognitive Psychology, pages 1–19, 2014.  doi:10.1093/oxfordhb/9780195376746.013.0010.
  * [46] J. Fuchs, P. Isenberg, A. Bezerianos, F. Fischer, and E. Bertini.  The influence of contour on similarity perception of star glyphs.  IEEE Computer Graphics and Applications, 20(12):2251–2260, 2014\.  doi:10.1109/TVCG.2014.2346426.
  * [47] K. Gadhave, J. Görtler, O. Deussen, M. Meyer, J. Phillips, and A. Lex.  Capturing User Intent when Brushing in Data Visualizations.  OSF Preprints, 2020.  URL: https://osf.io/mq2rk, doi:10.31219/osf.io/mq2rk.
  * [48] S. Galesso.  Instance segmentation with Mask R-CNN.  Technical report, Universität Freiburg, 2017.  URL: https://lmb.informatik.uni-freiburg.de/lectures/seminar_brox/seminar_ss17/maskrcnn_slides.pdf.
  * [49] M. Gall, G. Wallner, S. Kriglstein, and S. Rinderle-Ma.  A Study of Different Visualizations for Visualizing Differences in Process Models.  In Advances in Conceptual Modeling, pages 99–108. Springer International Publishing, 10 2015.  URL: http://link.springer.com/10.1007/978-3-319-25747-1_10, doi:10.1007/978-3-319-25747-1{\\_}10.
  * [50] A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, V. Villena-Martinez, P. Martinez-Gonzalez, and J. Garcia-Rodriguez.  A survey on deep learning techniques for image and video semantic segmentation.  Applied Soft Computing, 70:41–65, 2018.
  * [51] M. Ghoniem, J.-D. Fekete, and P. Castagliola.  On the readability of graphs using node-link and matrix-based representations: a controlled experiment and statistical analysis.  Information Visualization, 4(2):114–135, 2005.
  * [52] M. Girschick.  Difference detection and visualization in UML class diagrams.  Technical report, Technical University of Darmstadt, 2006.  URL: http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=CF8D78FE971562323D4D4353EE06DAB7?doi=10.1.1.88.3984&rep=rep1&type=pdf.
  * [53] R. Girshick.  Fast R-CNN.  In Proceedings of the 2015 IEEE International Conference on Computer Vision, pages 1440–1448. IEEE, 2015.  doi:10.1109/ICCV.2015.169.
  * [54] R. Girshick, J. Donahue, T. Darrell, and J. Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587. IEEE, 2014.
  * [55] M. Gleicher, D. Albers, R. Walker, I. Jusufi, C. D. Hansen, and J. C. Roberts.  Visual comparison for information visualization.  Information Visualization, 10(4):289—309, 2011.  doi:10.1177/1473871611416549.
  * [56] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  arXiv preprint arXiv:1406.2661, pages 1–9, 2014.
  * [57] D. Grattarola, D. Zambon, L. Livi, and C. Alippi.  Change Detection in Graph Streams by Learning Graph Embeddings on Constant-Curvature Manifolds.  IEEE Transactions on Neural Networks and Learning Systems, pages 1–14, 2019.  doi:10.1109/tnnls.2019.2927301.
  * [58] A. Grover and J. Leskovec.  Node2vec: Scalable feature learning for networks.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 855–864. Association for Computing Machinery, 2016.  doi:10.1145/2939672.2939754.
  * [59] K. Guckes.  Supplementary material for the dissertation “human similarity notion of graphs for visual comparison”.  Technical report, Technical University Darmstadt, 2022.  URL: http://www.projects.gris.informatik.tu-darmstadt.de/ballweg/intern/.
  * [60] K. Guckes (née Ballweg, M. Schä̈pers, M. Pohl, A. Kerren, and T. von Landesberger.  A shape change enhancing hierarchical layout for the pairwise comparison of directed acyclic graphs.  Technical report, Technical University Darmstadt, 2024.
  * [61] W. L. Hamilton, R. Ying, and J. Leskovec.  Representation Learning on Graphs: Methods and Applications.  Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 40(3):1–23, 2017.
  * [62] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik.  Simultaneous detection and segmentation.  In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors, Computer Vision – ECCV 2014, pages 297–312, Cham, 2014. Springer International Publishing.
  * [63] M. Hascoët and P. Dragicevic.  Interactive graph matching and visual comparison of graphs and clustered graphs.  In Proceedings of the International Working Conference on Advanced Visual Interfaces, pages 522—529. Association for Computing Machinery, 2012.  doi:10.1145/2254556.2254654.
  * [64] K. He, G. Gkioxari, P. Dollar, and R. Girshick.  Mask r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [65] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. IEEE, 2016.
  * [66] T. Hegeman and A. Iosup.  Survey of Graph Analysis Applications.  arXiv preprint arXiv:1807.00382, pages 1–23, 2018.
  * [67] P. Henderson and V. Ferrari.  End-to-end training of object class detectors for mean average precision.  In Asian Conference on Computer Vision, pages 198–213. Springer International Publishing, 2016.
  * [68] V. Heun, A. Von Kapri, and P. Maes.  Perifoveal display: Combining foveal and peripheral vision in one visualization.  UbiComp’12 - Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 1150–1155, 2012.
  * [69] I. U. Hewapathirana.  Change detection in dynamic attributed networks.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(3):1–20, 2019.
  * [70] M. Hlawatsch, M. Burch, and D. Weiskopf.  Visual adjacency lists for dynamic graphs.  IEEE Transactions on Visualization and Computer Graphics, 20(11):1590–1603, 2014.
  * [71] D. Holten, P. Isenberg, J.-D. Fekete, and J. J. van Wijk.  Performance evaluation of tapered, curved, and animated directed-edge representations in node-link graphs.  Technical report, INRIA, 2010.
  * [72] D. Holten, P. Isenberg, J. J. van Wijk, and J.-D. Fekete.  An extended evaluation of the readability of tapered, animated, and textured directed-edge representations in node-link graphs.  In 2011 IEEE Pacific Visualization Symposium, pages 195–202. IEEE, 2011.
  * [73] D. Holten and J. J. van Wijk.  A user study on visualizing directed edges in graphs.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2299–2308. Association for Computing Machinery, 2009\.  doi:10.1145/1518701.1519054.
  * [74] W. Huang, P. Eades, and S.-H. Hong.  Measuring effectiveness of graph visualizations: A cognitive load perspective.  Information Visualization, 8(3):139–152, 2009.  doi:10.1057/ivs.2009.10.
  * [75] W. Huang, Y. Qiao, and X. Tang.  Robust scene text detection with convolution neural network induced mser trees.  In European conference on computer vision, pages 497–511. Springer, 2014.
  * [76] M. Hussain, D. Chen, A. Cheng, H. Wei, and D. Stanley.  Change detection from remotely sensed images: From pixel-based to object-based approaches.  ISPRS Journal of Photogrammetry and Remote Sensing, 80:91–106, 2013\.
  * [77] E. Jamro and K. Wiatr.  Implementation of convolution operation on general purpose processors.  In Proceedings 27th EUROMICRO Conference. 2001: A Net Odyssey, pages 410–417. IEEE, 2001.
  * [78] S. Ji, Y. Shen, M. Lu, and Y. Zhang.  Building instance change detection from large-scale aerial images using convolutional neural networks and simulated samples.  Remote Sensing, 11(11), 2019.  doi:10.3390/rs11111343.
  * [79] P. Kazienko and T. Kajdanowicz.  Label-dependent node classification in the network.  Neurocomputing, 75(1):199–209, 2012.  doi:10.1016/J.NEUCOM.2011.04.047.
  * [80] D. Keim, G. Andrienko, J.-D. Fekete, C. Görg, J. Kohlhammer, and G. Melançon.  Visual Analytics: Definition, Process, and Challenges, pages 154–175.  Springer Berlin Heidelberg, 2008.  doi:10.1007/978-3-540-70956-5_7.
  * [81] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang.  On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima.  arXiv preprint arXiv:1609.04836, pages 1–16, 2017.
  * [82] S. Kieffer, T. Dwyer, K. Marriott, and M. Wybrow.  HOLA: Human-like orthogonal network layout.  IEEE Transactions on Visualization and Computer Graphics, 22(1):349–358, 2016.  doi:10.1109/TVCG.2015.2467451.
  * [83] T. N. Kipf and M. Welling.  Semi-Supervised Classification with Graph Convolutional Networks.  arXiv preprint arXiv:1609.02907, pages 1–14, 9 2016.
  * [84] A. Klippel, F. Hardisty, and C. Weaver.  Star plots: How shape characteristics influence classification tasks.  Cartogr. Geogr. Inf. Sci., 36(2):149–163, 2009.  doi:10.1559/152304009788188808.
  * [85] K. Kobayashi, J. Tsuji, and M. Noto.  Evaluation of data augmentation for image-based plant-disease detection.  In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2206–2211. IEEE, 2018.  doi:10.1109/SMC.2018.00379.
  * [86] G. Koch, R. Zemel, and R. Salakhutdinov.  Siamese neural networks for one-shot image recognition.  In Proceedings of the 32nd International Conference on Machine Learning, volume 37. JMLR.org, 2015.
  * [87] D. Koutra and C. Faloutsos.  Individual and Collective Graph Mining: Principles, Algorithms, and Applications.  Synthesis Lectures on Data Mining and Knowledge Discovery, 9(2):1–206, 2017.  doi:10.2200/s00796ed1v01y201708dmk014.
  * [88] C. Koylu and D. Guo.  Design and evaluation of line symbolizations for origin–destination flow maps.  Information Visualization, 16(4):309–331, 2017.  arXiv:https://doi.org/10.1177/1473871616681375, doi:10.1177/1473871616681375.
  * [89] S. Kriglstein, G. Wallner, and S. Rinderle-Ma.  A visualization approach for difference analysis of process models and instance traffic.  In F. Daniel, J. Wang, and B. Weber, editors, Business Process Management, pages 219–226. Springer Berlin Heidelberg, 2013.
  * [90] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84—90, 2017.  doi:10.1145/3065386.
  * [91] S. I. Ktena, S. Parisot, E. Ferrante, M. Rajchl, M. Lee, B. Glocker, and D. Rueckert.  Metric learning with spectral graph convolutions on brain connectivity networks.  NeuroImage, 169:431–442, 2018.
  * [92] A. Laurentini.  The visual hull concept for silhouette-based image understanding.  IEEE Transactions on pattern analysis and machine intelligence, 16(2):150–162, 1994.
  * [93] Y. LeCun and Y. Bengio.  Convolutional Networks for Images, Speech, and Time Series, pages 255–258.  MIT Press, 1998.
  * [94] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation Applied to Handwritten Zip Code Recognition.  Neural Computation, 1989.  doi:10.1162/neco.1989.1.4.541.
  * [95] B. Lee, C. Plaisant, C. S. Parr, J.-D. Fekete, and N. Henry.  Task taxonomy for graph visualization.  In Proceedings of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization, pages 1—5. Association for Computing Machinery, 2006.  doi:10.1145/1168149.1168168.
  * [96] K. Lenc and A. Vedaldi.  Understanding image representations by measuring their equivariance and equivalence.  International Journal of Computer Vision, 127(5):456—476, 2019\.  doi:10.1007/s11263-018-1098-y.
  * [97] M. Lima.  The book of trees: Visualizing branches of knowledge.  Princeton Architectural Press, 2014.
  * [98] T. Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie.  Feature pyramid networks for object detection.  In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pages 936–944. IEEE, 2017.  doi:10.1109/CVPR.2017.106.
  * [99] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors, Computer Vision – ECCV 2014, pages 740–755. Springer International Publishing, 2014.
  * [100] D. Liu, F. Guo, B. Deng, H. Qu, and Y. Wu.  egoComp: A node-link-based technique for visual comparison of ego-networks.  Information Visualization, 16(3):179–189, 2017.  doi:10.1177/1473871616667632.
  * [101] H. Liu, Z. Wang, F. Shang, M. Zhang, M. Gong, F. Ge, and L. Jiao.  A novel deep framework for change detection of multi-source heterogeneous images.  In 2019 International Conference on Data Mining Workshops (ICDMW), pages 165–171. IEEE, 2019.  doi:10.1109/ICDMW.2019.00034.
  * [102] J. Liu, K. Chen, G. Xu, H. Li, M. Yan, W. Diao, and X. Sun.  Semi-Supervised Change Detection Based on Graphs with Generative Adversarial Networks.  In IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, pages 74–77. IEEE, 2019.
  * [103] Y. Liu, C. Pang, Z. Zhan, X. Zhang, and X. Yang.  Building Change Detection for Remote Sensing Images Using a Dual Task Constrained Deep Siamese Convolutional Network Model.  arXiv preprint arXiv:1909.07726, pages 1–5, 2019.
  * [104] Z. Liu and J. Stasko.  Mental models, visual reasoning and interaction in information visualization: A top-down perspective.  IEEE Transactions on Visualization and Computer Graphics, 16(6):999–1008, 2010.  doi:10.1109/TVCG.2010.177.
  * [105] G. Ma, N. K. Ahmed, T. Willke, D. Sengupta, M. W. Cole, N. B. Turk-Browne, and P. S. Yu.  Similarity Learning with Higher-Order Graph Convolutions for Brain Network Analysis.  arXiv preprint arXiv:1811.02662, pages 1–11, 2018.
  * [106] Y. Ma, A. K. Tung, W. Wang, X. Gao, Z. Pan, and W. Chen.  ScatterNet: A Deep Subjective Similarity Model for Visual Analysis of Scatterplots.  IEEE Transactions on Visualization and Computer Graphics, 14(8):1–14, 2018.  doi:10.1109/TVCG.2018.2875702.
  * [107] N. H. Mackworth.  Stimulus density limits the useful field of view.  Eye movements and psychological processes, pages 307–321, 1976\.
  * [108] S. Marcel and Y. Rodriguez.  Torchvision the machine-vision package of torch.  In MM’10 - Proceedings of the ACM Multimedia 2010 International Conference, pages 1485–1488. Association for Computing Machinery, 2010.  doi:10.1145/1873951.1874254.
  * [109] I. Meirelles.  Design for information: an introduction to the histories, theories, and best practices behind effective information visualizations.  Rockport Publishers, 2013.
  * [110] G. Melancon.  Just how dense are dense graphs in the real world? a methodological note.  In Proceedings of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization, pages 1–7. Association for Computing Machinery, 2006.  doi:10.1145/1168149.1168167.
  * [111] K. Misue, P. Eades, W. Lai, and K. Sugiyama.  Layout adjustment and the mental map.  Journal of Visual Languages & Computing, 6(2):183–210, 1995.
  * [112] T. Moscovich, F. Chevalier, N. Henry, E. Pietriga, and J.-D. Fekete.  Topology-aware navigation in large networks.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2319–2328. Association for Computing Machinery, 2009\.
  * [113] T. Munzner.  Visualization Analysis and Design: Principles, Techniques, and Practice.  AK Peters Visualization Series. CRC Press, 2014.
  * [114] T. Munzner, F. Guimbretière, S. Tasiran, L. Zhang, and Y. Zhou.  Treejuxtaposer: Scalable tree comparison using focus+context with guaranteed visibility.  In ACM SIGGRAPH 2003 Papers, pages 453–462. Association for Computing Machinery, 2003.  doi:10.1145/1201775.882291.
  * [115] Q. H. Nguyen, H.-B. Ly, L. S. Ho, N. Al-Ansari, H. V. Le, V. Q. Tran, I. Prakash, and B. T. Pham.  Influence of data splitting on performance of machine learning models in prediction of shear strength of soil.  Mathematical Problems in Engineering, 2021:1–15, 2021.  doi:10.1155/2021/4832864.
  * [116] M. Niepert, M. Ahmed, and K. Kutzkov.  Learning convolutional neural networks for graphs.  In International conference on machine learning, pages 2014–2023, 2016.
  * [117] L. M. Padilla, S. H. Creem-Regehr, M. Hegarty, and J. K. Stefanucci.  Decision making with visualizations: a cognitive framework across disciplines.  Cognitive Research: Principles and Implications, 3(1):1–25, 2018\.  doi:10.1186/s41235-018-0120-9.
  * [118] H.-G. Pagendarm and F. H. Post.  Comparative visualization - approaches and examples.  Technical report, TU Delft, 1995.  LIDO-Berichtsjahr=1995,.  URL: https://elib.dlr.de/37368/.
  * [119] A. V. Pandey, J. Krause, C. Felix, J. Boy, and E. Bertini.  Towards understanding human similarity perception in the analysis of large sets of scatter plots.  In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 3659—3669. Association for Computing Machinery, 2016\.  doi:10.1145/2858036.2858155.
  * [120] E. Patuwo, M. Y. Hu, and M. S. Hung.  Two-Group Classification Using Neural Networks.  Decision Sciences, 24(4):825–845, 1993.  doi:10.1111/j.1540-5915.1993.tb00491.x.
  * [121] J. C. Peterson, J. T. Abbott, and T. L. Griffiths.  Adapting deep network features to capture psychological representations: An abridged report.  IJCAI International Joint Conference on Artificial Intelligence, pages 4934–4938, 2017.  doi:10.24963/ijcai.2017/697.
  * [122] T. Pflugshaupt, R. von Wartburg, P. Wurtz, S. Chaves, A. Déruaz, T. Nyffeler, S. von Arx, M. Luethi, D. Cazzoli, and R. M. Mueri.  Linking physiology with behaviour: Functional specialisation of the visual field is reflected in gaze patterns during visual search.  Vision Research, 49(2):237–248, 2009.  doi:10.1016/j.visres.2008.10.021.
  * [123] P. O. Pinheiro, R. Collobert, and P. Dollár.  Learning to segment object candidates.  arXiv preprint arXiv:1506.06204, pages 1–10, 2015.
  * [124] M. Pohl and A. Kerren.  Human Factors and Multilayer Networks.  In Workshop on Challenges in Multilayer Network Visualization and Analysis (Multilayer Nets ’19) at IEEE VIS’19, 2019.
  * [125] H. C. Purchase.  Which aesthetic has the greatest effect on human understanding?  In G. DiBattista, editor, Graph Drawing: 5th International Symposium, GD 1997, pages 248–261. Springer, 1997.  doi:10.1007/3-540-63938-1_67.
  * [126] H. C. Purchase.  Metrics for graph drawing aesthetics.  Journal of Visual Languages & Computing, 13(5):501–516, 2002.  doi:10.1006/S1045-926x(02)00016-2.
  * [127] H. C. Purchase, J. Hamer, M. Nöllenburg, and S. G. Kobourov.  On the usability of lombardi graph drawings.  In W. Didimo and M. Patrignani, editors, Graph Drawing, pages 451–462. Springer Berlin Heidelberg, 2013.
  * [128] H. C. Purchase, E. Hoggan, and C. Görg.  How important is the “mental map”? – an empirical investigation of a dynamic graph layout algorithm.  In M. Kaufmann and D. Wagner, editors, Graph Drawing: 14th International Symposium, GD 2006, pages 184–195. Springer, 2007.  doi:10.1007/978-3-540-70904-6_19.
  * [129] H. C. Purchase, C. Pilcher, and B. Plimmer.  Graph drawing aesthetics –
created by users, not algorithms.  IEEE Transactions on Visualization and
Computer Graphics, 18(1):81–92, 2012.  doi:10.1109/TVCG.2010.269.

  * [130] H. C. Purchase and A. Samra.  Extremes are better: Investigating mental map preservation in dynamic graphs.  In International Conference on Theory and Application of Diagrams, pages 60–73. Springer Berlin Heidelberg, 2008.
  * [131] R. J. Radke, S. Andra, O. Al-Kofahi, and B. Roysam.  Image change detection algorithms: a systematic survey.  IEEE Transactions on Image Processing, 14(3):294–307, 2005.  doi:10.1109/TIP.2004.838698.
  * [132] S. J. Raudys and A. K. Jain.  Small sample size effects in statistical pattern recognition: Recommendations for practitioners.  IEEE Transactions on Pattern Analysis & Machine Intelligence, 13(3):252–264, 1991.
  * [133] S. Ren, K. He, R. Girshick, and J. Sun.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, volume 39.  IEEE, 2017.
  * [134] E. C. Robinson, A. Hammers, A. Ericsson, A. D. Edwards, and D. Rueckert.  Identifying population differences in whole-brain structural networks: A machine learning approach.  NeuroImage, 50(3):910–919, 2010.  doi:10.1016/j.neuroimage.2010.01.019.
  * [135] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241. Springer International Publishing, 2015.
  * [136] S. Rufiange and M. J. McGuffin.  DiffAni: Visualizing Dynamic Graphs with a Hybrid of Difference Maps and Animation.  IEEE Transactions on Visualization and Computer Graphics, 19(12):2556–2565, 2013.  doi:10.1109/TVCG.2013.149.
  * [137] P. Saffrey and H. Purchase.  The “mental map” versus “static aesthetic” compromise in dynamic graphs: A user study.  In Proceedings of the Ninth Conference on Australasian User Interface - Volume 76, pages 85—93. Australian Computer Society, Inc., 2008\.
  * [138] B. Saket, D. Moritz, H. Lin, V. Dibia, C. Demiralp, and J. Heer.  Beyond Heuristics: Learning Visualization Design.  arXiv preprint arXiv:1807.06641, 2018.
  * [139] M. Schäpers.  An improved directed acyclic graph layout algorithm for visual comparison considering the directed acyclic graph’s shape.  Master’s thesis, Technical University Darmstadt, 2019.
  * [140] J. Schmidhuber.  Deep learning in neural networks: An overview.  Neural Networks, 61:85–117, 2015.  doi:10.1016/J.NEUNET.2014.09.003.
  * [141] P. Shilane, P. Min, M. Kazhdan, and T. Funkhouser.  The princeton shape benchmark.  In Proceedings of the Shape Modeling International 2004, pages 167—178. IEEE Computer Society, 2004.
  * [142] P. Spurek, T. Danel, J. Tabor, M. Smieja, L. Struski, A. Slowik, and L. Maziarka.  Geometric Graph Convolutional Neural Networks.  arXiv preprint arXiv:1909.05310, 2019.
  * [143] A. Srinivasan, M. Brehmer, B. Lee, and S. M. Drucker.  What’s the Difference?: Evaluating Variations of Multi-Series Bar Charts for Visual Comparison Tasks.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18, pages 1–12. Association for Computing Machinery, 2018.  doi:10.1145/3173574.3173878.
  * [144] H. Strasburger, I. Rentschler, and M. Jüttner.  Peripheral vision and pattern recognition: A review.  Journal of Vision, 11(5):1–82, 2011.
  * [145] K. Sugiyama, S. Tagawa, and M. Toda.  Methods for visual understanding of hierarchical system structures.  IEEE Transactions on Systems, Man, and Cybernetics, 11(2):109–125, 1981.
  * [146] J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei.  Line: Large-scale information network embedding.  In Proceedings of the 24th International Conference on World Wide Web, pages 1067–1077. International World Wide Web Conferences Steering Committee, 2015.  doi:10.1145/2736277.2741093.
  * [147] A. Telea.  Image-based graph visualization: Advances and challenges.  Graph Drawing and Network Visualization, pages 3–19, 2018.  doi:10.1007/978-3-030-04414-5{\\_}1.
  * [148] S. Teufel.  Evaluation.  Technical report, Natural Language and Inforamtion Prefecessing (NLIP) Group, University of Cambridge.  URL: https://www.cl.cam.ac.uk/~sht25/publications.html.
  * [149] P. Tittmann.  Graphentheorie, volume 2.  Hanser Fachbuchverlag, 2003.
  * [150] C. Tominski, C. Forsell, and J. Johansson.  Interaction support for visual comparison inspired by natural behavior.  IEEE Transactions on Visualization and Computer Graphics, 18(12):2719–2728, 2012.
  * [151] C. Treude, S. Berlik, S. Wenzel, and U. Kelter.  Difference computation of large models.  6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2007, pages 295–304, 2007.  doi:10.1145/1287624.1287665.
  * [152] A. Tversky.  Features of similarity.  In Psychological review, volume 84, page 327. American Psychological Association, 1977.
  * [153] A. Tversky.  Preference, belief, and similarity : selected writings.  MIT Press, 2004.
  * [154] C. Umbach and H. Gust.  Similarity-based classification.  In KogWis, 2010.
  * [155] A. C. Valdez, M. Ziefle, and M. Sedlmair.  Priming and Anchoring Effects in Visualization.  IEEE Transactions on Visualization and Computer Graphics, 24(1):584–594, 2018.  doi:10.1109/TVCG.2017.2744138.
  * [156] F. van Ham and B. Rogowitz.  Perceptual organization in user-generated graph layouts.  IEEE Transactions on Visualization and Computer Graphics, 14(6):1333–1339, 2008.
  * [157] A. Varghese, J. Gubbi, A. Ramaswamy, and P. Balamuralidhar.
changenet: A deep learning architecture for visual change detection.  In L.
Leal-Taixé and S. Roth, editors, Computer Vision – ECCV 2018 Workshops, pages
129–145. Springer International Publishing, 2019\.

  * [158] V. Verma and A. Pang.  Comparative flow visualization.  IEEE Transactions on Visualization and Computer Graphics, 10(6):609—624, 2004.  doi:10.1109/TVCG.2004.39.
  * [159] T. von Landesberger.  Visual Data Comparison.  Habilitation thesis, Technical University Darmstadt, 2017.
  * [160] T. von Landesberger, A. Kuijper, T. Schreck, J. Kohlhammer, J. J. van Wijk, J.-D. Fekete, and D. W. Fellner.  Visual analysis of large graphs: State-of-the-art and future research challenges.  Computer Graphics Forum, 30(6):1719–1749, 2011.  arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8659.2011.01898.x, doi:10.1111/j.1467-8659.2011.01898.x.
  * [161] T. von Landesberger, M. Pohl, G. Wallner, M. Distler, and K. Ballweg.  Investigating graph similarity perception: A preliminary study and methodological challenges.  In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP, (VISIGRAPP 2017), pages 241–250. SciTePress, 2017.  doi:10.5220/0006137202410250.
  * [162] E. Wall, L. M. Blaha, C. L. Paul, K. Cook, and A. Endert.  Four Perspectives on Human Bias in Visual Analytics, pages 29–42.  Springer International Publishing, 2018.  doi:10.1007/978-3-319-95831-6_3.
  * [163] G. Wallner, M. Pohl, C. Graniczkowska, K. Ballweg, and T. von Landesberger.  Influence of shape, density, and edge crossings on the perception of graph differences – an investigation under time constraints.  In Diagrammatic Representation and Inference, pages 348–356. Springer International Publishing, 2020.
  * [164] G. Wallner, M. Pohl, T. von Landesberger, and K. Ballweg.  Perception of Differences in Directed Acyclic Graphs: Influence Factors & Cognitive Strategies, pages 57–64.  Association for Computing Machinery, 2019.
  * [165] Q. Wang, Z. Chen, Y. Wang, and H. Qu.  Applying machine learning advances to data visualization: A survey on ml4vis.  arXiv preprint arXiv:2012.00467, pages 1–18, 2020.
  * [166] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [167] P. Warden.  How many images do you need to train a neural network?  Technical report, Jetpac, 2017.  URL: https://petewarden.com/2017/12/14/how-many-images-do-you-need-to-train-a-neural-network/.
  * [168] L. Wöhler, Y. Zou, M. Mühlhausen, G. Albuquerque, and M. Magnor.  Learning a Perceptual Quality Metric for Correlation in Scatterplots.  In Proceedings of the Vision, Modeling and Visualization (VMV). Eurographics, Eurographics Association, 2019.
  * [169] J. Wu, Y. Ye, Y. Chen, and Z. Weng.  Spot the difference by object detection.  arXiv preprint arXiv:1801.01051, 2018.
  * [170] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu.  A Comprehensive Survey on Graph Neural Networks.  arXiv preprint arXiv:1901.00596, pages 1–22, 2019.
  * [171] Y. Xie, M. Gong, S. Wang, W. Liu, and B. Yu.  Sim2vec: Node similarity preserving network embedding.  Information Sciences, 495:37–51, 2019.  doi:10.1016/J.INS.2019.05.001.
  * [172] K. Xu, W. Hu, J. Leskovec, and S. Jegelka.  How Powerful are Graph Neural Networks?  arXiv preprint arXiv:1810.00826, pages 1–17, 2018.
  * [173] K. Xu, C. Rooney, P. Passmore, and D.-H. Ham.  A user study on curved edges in graph visualisation.  In P. Cox, B. Plimmer, and P. Rodgers, editors, Diagrammatic Representation and Inference, pages 306–308. Springer Berlin Heidelberg, 2012\.
  * [174] V. Yoghourdjian, D. Archambault, S. Diehl, T. Dwyer, K. Klein, H. C. Purchase, and H.-Y. Wu.  Exploring the limits of complexity: A survey of empirical studies on graph visualisation.  Visual Informatics, 2(4):264–282, 2018.  doi:https://doi.org/10.1016/j.visinf.2018.12.006.
  * [175] V. Yoghourdjian, Y. Yang, T. Dwyer, L. Lawrence, M. Wybrow, and K. Marriott.  Scalability of network visualisation from a cognitive load perspective.  IEEE Transactions on Visualization and Computer Graphics, 2020.
  * [176] A. Zalesky, A. Fornito, and E. T. Bullmore.  Network-based statistic: Identifying differences in brain networks.  NeuroImage, 53(4):1197–1207, 2010.  doi:10.1016/J.NEUROIMAGE.2010.06.041.
  * [177] L. Zaman, A. Kalra, and W. Stuerzlinger.  The effect of animation, dual view, difference layers, and relative re-layout in hierarchical diagram differencing.  In Proceedings of Graphics Interface 2011, pages 183—190. Canadian Human-Computer Communications Society, 2011.
  * [178] M. D. Zeiler.  Adadelta: an adaptive learning rate method.  arXiv preprint
arXiv:1212.5701, pages 1–6, 2012.

  * [179] M. D. Zeiler and R. Fergus.  Visualizing and Understanding Convolutional Networks.  In Computer Vision – ECCV 2014, pages 818–833. Springer International Publishing, 2014.  doi:10.1007/978-3-319-10590-1{\\_}53.
  * [180] G. Zhang, A. P. Auchus, P. Kochunov, N. Elmqvist, and J. Chen.  Overlaying Quantitative Measurement on Networks: An Evaluation of Three Positioning and Nine Visual Marker Techniques.  arXiv preprint arXiv:1711.04419, pages 1–14, 2017.
  * [181] S. Zhang, H. Tong, J. Xu, and R. Maciejewski.  Graph convolutional networks: a comprehensive review.  Computational Social Networks, 6(1):11, 2019.  doi:10.1186/s40649-019-0069-y.
  * [182] Z. Zhang, P. Cui, and W. Zhu.  Deep learning on graphs: A survey.  IEEE Transactions on Knowledge and Data Engineering, pages 1–20, 2020.
  * [183] T. Zhou, L. Lü, and Y.-C. Zhang.  Predicting missing links via local information.  The European Physical Journal B, 71(4):623–630, 2009.  doi:10.1140/epjb/e2009-00335-8.
  * [184] B. Zhu, H. Gao, X. Wang, M. Xu, and X. Zhu.  Change Detection Based on the Combination of Improved SegNet Neural Network and Morphology.  In 2018 IEEE 3rd International Conference on Image, Vision and Computing (ICIVC), pages 55–59. IEEE, 2018.
  * [185] Z. Zou, Z. Shi, Y. Guo, and J. Ye.  Object Detection in 20 Years: A Survey.  arXiv preprint arXiv:1905.05055, pages 1–39, 2019.
  * [186] D. Zuo, X. Liu, and S. Zhu.  Application of affine transformation in traffic sign detection.  In 2010 International Conference on Intelligent Control and Information Processing, pages 277–280. IEEE, 2010.  doi:10.1109/ICICIP.2010.5565311.