Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.

The work of BSS at the University of Virginia was partially funded by the NSF
under grant DMR #2016909.

Corresponding author: Vinícius Yu Okubo (e-mail: ViniciusOkubo@usp.br).

# Characterization of Magnetic Labyrinthine Structures through Junctions and
Terminals Detection using Template Matching and CNN

VINíCIUS YU OKUBO1  KOTARO SHIMIZU2, B. S. SHIVARAM3, HAE YONG KIM1  Dept.
Electronic Systems Engineering, Polytechnic School, University of São Paulo,
Brazil  Department of Applied Physics, The University of Tokyo, Tokyo
113-8656, Japan  Department of Physics, University of Virginia,
Charlottesville, Virginia 22904, USA

###### Abstract

In material sciences, characterizing faults in periodic structures is vital
for understanding material properties. To characterize magnetic labyrinthine
patterns, it is necessary to accurately identify junctions and terminals,
often featuring over a thousand closely packed defects per image. This study
introduces a new technique called TM-CNN (Template Matching - Convolutional
Neural Network) designed to detect a multitude of small objects in images,
such as defects in magnetic labyrinthine patterns. TM-CNN was used to identify
these structures in 444 experimental images, and the results were explored to
deepen the understanding of magnetic materials. It employs a two-stage
detection approach combining template matching, used in initial detection,
with a convolutional neural network, used to eliminate incorrect
identifications. To train a CNN classifier, it is necessary to create a large
number of training images. This difficulty prevents the use of CNN in many
practical applications. TM-CNN significantly reduces the manual workload for
creating training images by automatically making most of the annotations and
leaving only a small number of corrections to human reviewers. In testing, TM-
CNN achieved an impressive F1 score of 0.988, far outperforming traditional
template matching and CNN-based object detection algorithms.

###### Index Terms:

Computer vision, convolutional neural networks, deep learning, magnetic
labyrinthine patterns, material science, object detection, template matching.

=-21pt

##  I Introduction

In materials, a wide range of periodic structures are seen accompanied by
defects within their spatial arrangement. Some magnetic materials exhibit
stripe orders, wherein the orientation of electron spins i.e. the magnetic
moments show periodic alterations [1], as well as labyrinthine structures,
characterized by magnetic arrangements resulting from the propagation of
stripes in various directions throughout space [2, 3, 4]. The two images in
Fig. I are representative examples showcasing labyrinthine patterns of
magnetic domains in Bismuth-doped Yttrium Iron Garnet (Bi:YIG) films at zero
field; the intensity represents the out-of-plane component of the magnetic
moments. These labyrinthine patterns may present discernible characteristics
that are difficult to quantify. As shown in Fig. Ia, which we label as the
“quenched” state, the border of dark and bright domains exhibit a sinuous
nature and do not appear as parallel. In contrast, the “annealed” state, shown
in Fig. Ib, consists of regions with nearly parallel domains. This state
exhibits roughly equal widths of dark and bright domains, and the areas
occupied by them are also approximately equal for any sampled region.
Therefore, the stripes in the annealed state show greater spatial coherence.

Within these magnetic structures, defects take the form of interruptions in
the stripes known as “terminals” and points where multiple stripes conjoin,
referred to as “junctions” (Fig. 1). The presence of defects not only serves
as a crucial metric for quantifying the deviation of a structure from a
perfectly periodic structure, but has also gathered considerable attention in
recent years due to its implications on physical phenomena arising from the
nontrivial geometric properties associated with magnetic structures [5]. Thus,
in the realm of condensed matter physics, experimental identification of the
number and positions of such defects plays an important role in characterizing
material properties.

[h](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.45]okubo1.png Examples of
magnetic labyrinthine patterns.

Accurately counting and differentiating genuine structures from
misidentifications is crucial to a quantitative physical understanding of the
origins and evolution of these patterns. Manual annotation of defects in
unfeasible. For instance, we used 444 images with 641,649 structures [6].
Furthermore, manual annotation relies on subjective interpretation of
junctions and terminals, which could lead to counting inconsistencies. In
order to address these issues, an automated process is required. Here,
algorithms for finding objects in an image known as object detectors are an
excellent choice and can be very effective. They can be broadly categorized
into classical and deep learning-based methods.

###  I-A Classical object detection methods

Classical object detection methods span a variety of techniques to extract and
process features from the image. For instance, template matching [7] is a
technique employed to find a pre-defined template within a larger image. This
is achieved by scanning the entire image and calculating the correlation
between the template and the scanned region. Viola and Jones object detecting
algorithm [8] employs multiple template matchings using Haar-like features,
each of them serving as a weak detector. These features are ensembled through
a boosting strategy to form a strong detector. The Histogram of Oriented
Gradients (HOG) [9] represents another approach to extract useful features,
dividing the image into cells and calculating a gradient histogram for each.
This extracted information can be used by machine learning algorithms, such as
support vector machine, to identify detections.

###  I-B Deep learning object detection methods

In the last decade, deep learning approaches have surpassed traditional
machine learning techniques in multiple image processing tasks [10, 11]. For
object detection, Girshick et al. introduced R-CNN [12], marking it as one of
the pioneering detection techniques rooted in deep learning. R-CNN operates as
a classification-based model: multiple regions are extracted from an image and
each is classified independently. This straightforward method led to
significant improvements in detection, achieving new state-of-the-art results
in the Pascal VOC dataset [13], compared to earlier techniques like Haar
features and HOG. A distinguishing feature of R-CNN is its region proposal
step. Directly processing every conceivable region of varying sizes and
positions in an image is computationally impractical. Hence, this step selects
a simplified set of regions from the original image for individualized
classification by the CNN model. Further developments brought by Faster R-CNN
[14] have improved both accuracy and speed by integrating the region proposal
into the model.

Redmon et al. introduced YOLO [15], a deep-learning detection approach modeled
as a regression task. This method partitions the image into a grid and each
cell contain their own set of outputs. The grid cell where the object is
centered has the task of identifying the position, dimensions and class of the
object. A standout benefit of this approach is its efficiency: YOLO processes
the image in a single pass, contrasting with R-CNN-based models that are
divided into region proposal and classification steps. This significantly
reduces inference time, enabling real-time video detection. However, YOLO was
not able to achieve the precision and recall rates of Faster R-CNN when tested
on the Pascal VOC dataset. Comparisons in small object detection settings have
also shown Faster R-CNN to outperform YOLO [16].

Figure 1: Defects in the magnetic labyrinth patterns. Junctions are marked in
green and terminals in cyan. Red dots indicate false positives from the
template matching, which are filtered out by the CNN classifier.

###  I-C The problem and proposed solution

Modern digital microscopy allows the easy acquisition of high-resolution
experimental images, which contain intricate stripe patterns and a large
number of defects, sometimes numbering in the thousands. This makes it
difficult to use deep learning, as creating an accurately annotated dataset
would be a laborious process due to the large number of defects. Furthermore,
both YOLO and Faster R-CNN were not designed to detect thousands of small
objects. Benchmarks with both methods show that performance degrades when
detecting small objects, with Faster R-CNN maintaining a slight advantage over
YOLO [16].

Correlation-based granulometry is a technique that can be used to detect a
large number of small objects in the image. It was proposed by Maruta et al.
[17, 18] to analyze the distribution of square and circular pores in the
macroporous silicon layer in scanning electron microscope images. This
technique is called “granulometry” because the objective of the original
application was to obtain a histogram of pore distribution as a function of
size. It was later used by Araújo et al. [19] to detect individual bean
grains, analyze each grain, and calculate the quality of the bean batch. This
technique performs multiple template matchings to achieve robustness against
angle and shape variations, and then performs non-maximum suppression to avoid
finding the same object multiple times. However, this technique could not be
applied directly in our problem, because defects in labyrinthine magnetic
structures vary greatly in shape and cannot be accurately detected using only
template matchings.

To address these challenges, we developed a new two step technique that we
named Template Matching - Convolutional Neural Network (TM-CNN), which
integrates correlation-based granulometry with the CNN classifier. First, a
series of template matchings detects potential candidates for junctions and
terminals. By setting a low threshold, we cast a broad net for candidates
detection, minimizing false negatives but increasing false positives. These
candidates are filtered by non-maximum suppression to avoid multiple
detections of the same defect. Second, a CNN classifier filters out the false
positives.

We demonstrate that this method substantially outperforms template matching
alone, while streamlining the image annotation process and reducing the
computational burden typically associated with deep learning detection
techniques. We also show that the TM-CNN technique outperforms Faster R-CNN,
achieving a significantly higher F1-score in junctions and terminals
detection.

###  I-D Structure of the article

The remainder of this work is organized as follows. Section II presents
existing techniques for detecting junctions and terminals, differentiating
classical and deep learning approaches. Section III describes the dataset used
and the proposed TM-CNN technique in detail. Section IV presents experimental
results from the detection of defects in magnetic labyrinthine patterns and
discusses their significance in understanding physical phenomena. Finally,
Section V concludes the article by reflecting on the merits of our work.

##  II Related Works

###  II-A Junctions and terminals detection

Junctions and terminals are shapes with relevance that extend beyond
materials. Within computer vision, recognizing and enumerating them has been
performed in diverse contexts, such as natural landscapes [20], biology [21]
and handwriting images [22].

###  II-B Classical methods for junctions and terminals detection

One prevalent approach for detecting junctions and terminals involves using
skeletonization as a pre-processing step. This process reduces the image to
one-pixel-wide lines to represent its structures. Points in skeleton can then
be identified as terminals, junctions and crossings based on their neighboring
pixels. This technique has been applied in vascular images [23] and in the
analysis of handwritten Chinese characters [24].

Pre-processing techniques using contour information have also been explored
for junction detection. Lee and Wu [25] investigated stroke extraction in
Chinese characters. Their method segments regions according to their contour,
identifying junctions by counting neighboring regions. Maire et al. [26]
proposed a junction detector in natural images by locating intersecting
contours. This approach applies an expectation–maximization style algorithm to
iteratively select relevant contours and suggest the junction’s position.

However, skeletonization and contour finding are noise-sensitive processes and
a pre-processing error will lead to a detection error.

Junctions and terminals can also be identified by analyzing the arrangement of
linear structures within the image. Su et al.[27] describe a technique for
identifying these linear structures using the Hessian Matrix. This approach
has been validated in biological images such as blood vessels, neutrites and
tree branches. Xia et al. [20] present a junction detection method in natural
images, based on amplitudes and phases of the normalized gradients of the
image.

Template-based approaches quantify the similarity of the appearance of image
regions and the template. Deriche and Blaszka [28] modeled this approach as
energy minimization, which is calculated by the deviation between the image
and a predetermined model. This enabled the detection of key image features,
such as edges, corners and terminals.

###  II-C Deep learning methods for junctions and terminals detection

Owing to the success of R-CNN based detection techniques, Pratt et al. [21]
developed a pipeline for identifying junctions and crossings in retinal
vascular structures. Their method involves two main steps: initially,
detection regions are proposed centred along the blood vessels, which are then
classified as junctions, crossings or background. To generate the detection
regions, their approach requires a binary segmented version of the exam.
Theses images undego a skeletonization process, with the resulting points
serving as references for the centers of the blood vessels.

Zhao et al. [29], addressing the same problem, proposed using a Mask R-CNN
base model[30] for region proposal. This strategy enables inference without
the need for binary segmented version of the exam. However, during training,
the segmented images are still used in the Mask R-CNN model to enhance its
learning capabilities. This approach surpassed the performance of previous
techniques in the detection of junctions and crossings in retinal vascular
images.

##  III Methodology

###  III-A Dataset

In this study, we used films of a ferromagnetic material of recognized
technological importance, Bi:YIG, and obtained magnetic images of the
labyrinthine patterns using a microscope with polarized light [31]. We
specifically focused on the evolution of the labyrinthine patterns under the
demagnetization field protocol described below. First, we prepared the sample
in the fully saturated state by applying a sufficiently large magnetic field
in $+z$ direction, which is perpendicular to the films. In this state,
magnetic moments in the Bi:YIG film are forced to point in the field
direction. Next, we instantaneously dropped the field to zero and hold it for
10 seconds to get the image. Since the magnetic field is zero, magnetic
moments can point upward and downward, resulting in the labyrinthine patterns
shown in Fig. I; the bright and dark regions represent the domains with
opposite directions of magnetic moments. Such a process involving switching on
and off the magnetic fields is considered half of the demagnetization step. In
the remaining half step, a magnetic field was applied with a reduced amplitude
and oriented in the opposite direction. We again captured the magnetic domain
image after reducing the magnetic field to zero. By repeating these protocols
up to 18 steps, we investigated the evolution of the labyrinthine patterns in
the demagnetization process step by step. The amplitude of the magnetic field
was exponentially reduced with each step. We conducted a series of
demagnetization processes from the fully saturated state, repeating this cycle
six times. Furthermore, we explored another situation, where the magnetic
field was initially applied in the $-z$ direction, and its direction was
alternated step by step. Consequently, a total of 12 demagnetization processes
were performed, yielding a collection of 444 domain images. All measurements
reported here were performed at room temperature. The experimentally obtained
images covered an area of 2 mm $\times$ 1.8 mm.

The original high-resolution color images were converted to grayscale and
their resolutions were reduced to 1300×972 to facilitate processing.
Furthermore, a median filter with kernel size 3 was applied to reduce noise.

###  III-B TM-CNN overview

Our approach to detect junctions and terminals in magnetic labyrinthine
patterns consists of two sequential steps: proposal of potential detections
and their classification between junction, terminal and false detections. It
is inspired by other cascaded object detection techniques like Viola and Jones
face detection [8], R-CNN [12], and scale and rotation invariant template
matching [32].

Fig. 2 illustrates the overall structure of TM-CNN. In the first phase, the
algorithm generates a preliminary set of potential detections. It must propose
all true defects, even if it also generates many false positives. We achieve
this by applying template matching detection with a low threshold, followed by
a non-maximum suppression. In the second phase, in order to eliminate the
false positives, each potential detection is filtered by a CNN classifier.

Figure 2: TM-CNN overview: (a) Our technique receives an image of magnetic
labyrinthine patterns as input. (b) It applies template matchings to compute
correlation maps. (c) Correlation maps measure the similarity between the
templates and each region of the image. (d) The algorithm looks for high
values in the correlation maps and applies a breadth-first search for non-
maximum suppression. (e) The centers of potential detections are located at
the highest correlation values inside the searched area. (f) The algorithm
extracts 50×50 patches centered on each potential detection and classifies
them using a CNN. (g) The classifier determines whether the detection is a
junction, a terminal or a false detection.

###  III-C Template matching

The basic form of template matching finds instances of a smaller template $T$
within a larger image $I$. This is done by calculating some similarity metric
between the model $T$ and the content of a moving window located at each
possible position of $I$. We measured the similarity between the image and the
template using the Normalized Cross Correlation (NCC). NCC is invariant to
linear changes in brightness and/or contrast. NCC between template $T$ and
image $I$ at pixel $(x,y)$ is calculated as:

$\begin{split}NCC_{T,I}&(x,y)=\\\
&\frac{\sum_{x^{\prime},y^{\prime}}(\tilde{T}(x^{\prime},y^{\prime})\cdot\tilde{I}(x+x^{\prime},y+y^{\prime}))}{\sqrt{\sum_{x^{\prime},y^{\prime}}\tilde{T}(x^{\prime},y^{\prime})^{2}\cdot\sum_{x^{\prime},y^{\prime}}\tilde{I}(x+x^{\prime},y+y^{\prime})^{2}}}\end{split}$
(1)

where $(x^{\prime},y^{\prime})$ is the coordinate inside the template,
$\tilde{T}(x^{\prime},y^{\prime})$ is the mean-corrected template value at
$(x^{\prime},y^{\prime})$, and $\tilde{I}(x,y)$ is the value of the mean-
corrected image inside the moving window at pixel $(x,y)$. Mean-correction
consists of subtracting the average value from each pixel. Template matching
using NCC can be implemented very efficiently using Fast Fourier Transform
(FFT) [33].

This basic approach is not well suited for detecting junctions and terminals
in the magnetic labyrinthine structures because a single template is not
capable of modeling:

  1. 1.

All possible rotations;

  2. 2.

All deformed shapes of defects.

To solve problem (1), we employ a rotation-invariant template matching based
on exhaustive evaluation of rotated templates. There are some alternative
rotation-invariant techniques based on circular and radial projections [34,
32], and on Fourier coefficients of circular and radial projections [35].
These techniques can reduce computational requirements, but their
implementations are complex and require parameter tuning. Furthermore, our
application does not require exceptional computational performance, as the
processing is offline. Thus, our technique uses the standard OpenCV111Open
Source Computer Vision Library, https://opencv.org template matching
implementation, which is highly optimized using FFT and special processor
instructions.

Problem (2) arises from complex shape variations of defects in magnetic
labyrinthine patterns. Typically, terminals and junctions do not align
perfectly with the templates, mainly because real magnetic strips have non-
uniform curvatures and widths. Template matching allows for slight shape
variations. However, to further improve flexibility, we use masks. Many
template matching implementations allow you to specify a mask, besides the
template, indicating which pixels in the template will not be considered in
the correlation calculation. White pixels in the mask indicate significant
template pixels, and black pixels indicate template pixels to disregard
(“don’t care”). If the mean-corrected template value at some pixel
$(x^{\prime},y^{\prime})$ is zero, that is
$\tilde{T}(x^{\prime},y^{\prime})=0$, this pixel will not be taken into
account to compute NCC. On the other hand, one cannot use FFT to speed up the
NCC calculation if it contains conditional executions. Therefore, it is very
likely that optimized implementations will assign zero to the template pixels
corresponding to the black mask pixels, after the mean correction.

###  III-D Templates and masks used in the experiment

We manually designed templates and masks, and empirically tuned them to
capture various junction and terminal shapes. They have 21×21 resolutions and
are generated at runtime. Templates represent magnetic strips as black lines
radiating from their center drawn on a white background. Meanwhile, masks are
created from black backgrounds with white areas defining relevant coordinates
used in template matching. Their main purpose is to obscure the space between
strips and background, addressing variations in widths and curvatures, and to
limit the background regions to reduce interference from neighboring strips.
Table I exemplifies the templates and masks used to detect junctions and
terminals.

We created 439 junction templates to accommodate variations in shapes and
rotations. Junction templates consist of three radial lines at rotation
$\alpha,\beta,\gamma\in\mathbb{N}$ with
$0^{\circ}\leq\alpha<\beta<\gamma<360^{\circ}$ (Table I). These templates are
drawn iteratively in steps of 15°, while maintaining a difference of at least
70° and at most 190° between two consecutive angles. For each triple of
parameters $(\alpha,\beta,\gamma)$, a single template is generated along with
three masks: two masks “type A” where the effective background is parallel to
strips and one mask “type B” where the effective background is positioned
between the strips (Table II). Masks are created according to the rotations
$\alpha,\beta,\gamma$ and width values $w_{bg},w_{bs},w_{sp}\in\mathbb{N}$,
representing the width of the effective background, the width between
effective background and strip, and the width of the effective strip,
respectively (Table I).

TABLE I: Templates and masks used to detect junctions and terminals |  Template |  Mask |  Mask description
---|---|---|---
Junction |  |  |
Terminal |  |  |
Effective strip  Effective background  Discarded region

Terminal templates are simpler and use a single rotation parameter
$\alpha\in\mathbb{N}$, varying within $0^{\circ}\leq\alpha<360^{\circ}$ in
steps of 3°, totaling 120 variants. To model variations in terminal and
background shapes, we created five masks for each model. Terminal masks are
determined by six parameters, including the previous four
$(\alpha,w_{bg},w_{bs},w_{sp})$, and the two additional radius parameters
$R_{bs},R_{bg}$ that define the two circles at the tip of the terminal,
reflecting the strip and background respectively.

TABLE II: Collection of masks used in our study |  $w_{bg}$ |  $w_{bs}$ |  $w_{sp}$ |  $R_{bg}$ |  $R_{bs}$ |  Mask
---|---|---|---|---|---|---
Junction  ${}_{(\alpha=60^{\circ},\beta=180^{\circ},\gamma=300^{\circ})}$
junction_mask 1 (type A) |  1 |  7 |  3 |  - |  - |
junction_mask 2 (type A) |  3 |  5 |  2 |  - |  - |
junction_mask 3 (type B) |  -6 |  6 |  2 |  - |  - |
Terminal  ${}_{(\alpha=30^{\circ})}$
terminal_mask 1 |  2 |  3 |  2 |  5 |  2 |
terminal_mask 2 |  1 |  5 |  2 |  5 |  3 |
terminal_mask 3 |  0 |  4 |  2 |  7 |  4 |
terminal_mask 4 |  1 |  6 |  2 |  10 |  8 |
terminal_mask 5 |  0 |  7 |  2 |  8 |  4 |

Template matching is applied separately for each (template, mask) pair.
Several template matchings are computed in parallel using the OpenMP
library222Open Multi-Processing, https://www.openmp.org/. This process takes
about 80 seconds to process an image on an i7-9750H processor.

To obtain the final correlation map $corr$, we calculate the maximum value
among all $n=3\times 439+5\times 120=1917$ NCC maps for each position $(x,y)$,
that is:

$corr(x,y)=\textup{MAX}_{i=1}^{n}\bigr{[}\textup{NCC}_{i}(x,y)\bigr{]}$ (2)

Pixels where $corr(x,y)$ exceeds a predefined threshold $t$ are considered
potential detections.

However, a single junction/terminal may encompass multiple neighboring points
with correlation values greater than the threshold $t$. Therefore, it is
necessary to perform some form of non-maximum suppression to eliminate
duplicate detections and select only the true center of junction/terminal. Kim
et al. [18] present a solution to this problem: Whenever two potential
detection points $p_{1}$ and $p_{2}$ are separated by a distance smaller than
a threshold $c$, the point with the lowest correlation value is discarded. In
this work, we use a slightly different approach, but with the same practical
result: Whenever the algorithm finds a potential detection, it executes a
breadth-first search algorithm. This algorithm recursively searches adjacent
pixels $(x,y)$ where the correlation exceeds 80% of the threshold (that is,
$corr(x,y)>0.8t$) and saves the pixel with the highest correlation.
Subsequently, the searched area has its correlation value set to zero to avoid
re-detection. Fig. 2d highlights the searched area in red. The pixel with the
highest correlation is chosen as the center of the junction/termination. This
process performs detection in a single pass.

###  III-E Dataset annotation

To classify potential detections into true or false using a CNN classifier, we
must first create annotated training images. TM-CNN makes it easier to create
training examples, as it allows one to annotate the examples semi-
automatically. This process is divided in two phases.

####  III-E1 Template matching-assisted annotation

In this phase, only a small set of images are annotated. Initially, we apply
template matching followed by non-maximum suppression to identify the centers
of possible detections, together with their probable labels (junction or
terminal). Without this help from template matching, we would have to manually
and precisely locate the centers of thousands of defects. Subsequently, a
human reviewer makes corrections to ensure that the labels given by the
template matching are correct. The reviewer may change the labels to junction,
terminal or false detection. After all positive detections are annotated along
with a small set of false detections, a larger set of false detections is
created by lowering the template matching threshold and sampling new false
detections. These images, now with positive and negative annotations, are used
to train a preliminary version of the CNN classifier.

####  III-E2 Deep learning-assisted annotation

Due to the small number of images in the initial training set, the preliminary
CNN classifier cannot accurately classify all magnetic stripe defects.
Nonetheless, this preliminary model is integrated into the annotation
procedure to alleviate the required workload. In the second phase, we continue
using template matching to generate the initial set of detections. However,
the preliminary CNN classifier is employed to identify most of the template
matching errors, thus speeding up the annotation process. As new images are
annotated, more accurate models are trained to further simplify the annotation
workload. The final training set consists of 17 images derived from a single
annealing protocol, selected to cover varied experimental configurations of
ascending and descending magnetic fields at different magnitudes. Out of
these, 16 were selected from the quenched (unordered) state, as they represent
a more diverse set of shapes and represent a greater challenge for
classification. The training set encompasses a total of 33,772 detections,
which includes 12,144 junctions, 12,777 terminals, and 8,851 false detections.

###  III-F Candidate filtering by CNN

Our algorithm extracts small 50×50 patches centered around each detection
point and a CNN classifies them into three classes: junction, terminal or
false positive. The size of patches for CNN classification is larger than the
size of template matching models (21×21), allowing CNN to use more contextual
information than template matching.

We use a simple CNN model to classify the small patches (Fig. III-F). It has
four convolutional layers with 32, 64, 128 and 256 filters, all using 3×3
kernels. The first three convolutional layers are followed by max pooling
layers to downsample the feature maps and global max pooling is applied after
the last convolutional layer. This is followed by dropout and two fully
connected layers: the first with 128 nodes and the second with three output
nodes. The ReLU activation function is used across the model, except at the
output layer where the softmax is used. In total, this network has only
422,608 parameters. For comparison, VGG-16 and ResNet-50 (common backbones for
detection) have 138 million and 23 million parameters, respectively. Thanks to
its simplicity, our model is fast and can make predictions even without GPUs
and takes around 30 seconds for filtering each image using an i7-9750H
processor with 16 GB of RAM.

[h](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.45]okubo23.png CNN
architecture to classify a patch in junction, termination or false detection.

We implemented this model in Python using the TensorFlow
library333https://www.tensorflow.org/. To enhance rotational invariance, we
employed data augmentation techniques, performing random rotations of the
images in multiples of $90^{\circ}$. We pre-trained it using the MNIST [36]
dataset for 15 epochs. We then fine-tuned the model for our application for
another 20 epochs. We used categorical cross-entropy as the loss function, and
the model was optimized using Adam with the learning rate of $10^{-3}$.

##  IV Experiments and Results

###  IV-A Performance evaluation

Evaluating TM-CNN poses a certain challenge because there are no benchmarks or
published results for direct comparison. So, we defined that junction and
terminal detections reviewed by human observers as the gold standard and
compared the algorithms against this standard. For testing purposes, we
annotated 5 independent images from varying experiments, using the same
procedure as the training images (subsection III-E). False detections were
excluded since they do not contribute to the detection evaluation. In total,
the test set contained 3,551 junctions and 3,653 terminals.

We tested four algorithms: template matching only (our TM-CNN algorithm,
without CNN from the second stage); the proposed TM-CNN algorithm; the widely
used Faster R-CNN; and its variant $CLS_{Conv4}$ proposed by Eggert et al.
that improves small object detection [37]. The Faster R-CNN model had to be
adapted, since it was originally designed to detect large objects. We used a
single anchor of size 24×24, as all defects are approximately this size. The
model’s backbone was VGG16, in line with Ren et al.[14]. This backbone was
initially pre-trained on the MNIST dataset for digit classification. We fine-
tuned the model for junction and terminal detection for 25 epochs using the 17
training images, with learning rate of $10^{-5}$ and the Adam optimizer.

We used precision, recall and F1-score as evaluation metrics. A detection was
considered positive if it achieved Intersection over Union (IoU) greater than
0.5, and the detection threshold was adjusted to optimize the F1-score.
Results presented in Table III show that TM-CNN substantially outperforms
template matching only, standard Faster R-CNN and its variant $CLS_{Conv4}$.
Our algorithm achieves detection accuracy close to 100% and most mistakes are
in detections where a human observer would also be in doubt about the correct
classification (Fig. IV-A).

TABLE III: Performance of different algorithms to detect junctions and terminals Model |  Precision |  Recall |  F1-Score
---|---|---|---
Template matching only |  0.838 |  0.935 |  0.883
TM-CNN |  0.986 |  0.990 |  0.988
Faster R-CNN |  0.607 |  0.330 |  0.427
$CLS_{Conv4}$ [37] |  0.761 |  0.649 |  0.701

[h](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.45]okubo24.png Examples of
TM-CNN detection errors, highlighted in red. Green and cyan dots are
respectively junctions and terminals detected by TM-CNN.

###  IV-B Results from a Physics perspective

In Bi:YIG films, it is known that a perfect stripe pattern is the most
energetically favorable and stable configuration of magnetic moments [4]. In
this structure, the magnetic moments exhibit stripes aligning straight in one
direction throughout the entire structure, resulting in large spatial
coherence. Meanwhile, the experimentally obtained structures often manifest
labyrinthine patterns, as illustrated by dark and bright stripes in Fig. I. In
these labyrinthine patterns, stripes are aligned within small domains but
propagate in different directions between different domains, and thus the
spatial coherence of stripes is smaller than the perfect stripe. Notably,
junctions and terminals are defects emerging in the formation of these
labyrinthine patterns, intricately linked to the growth of spatial coherence
in the system. Introducing such defects into the perfect stripe structure
leads to the bending and branching of the initially aligned stripes,
eventually reducing spatial coherence accompanied by the transformation to
labyrinthine patterns. Indeed, a plethora of defects are observed in the
labyrinthine patterns shown in Fig. 1. Conversely, the elimination of these
defects plays a crucial role in the transition from a quenched state to an
annealed state, where the system increases the spatial coherence.

Each of these defects cannot be eliminated alone by continuous deformation of
the stripes due to their topological properties. They can be removed through
the pair annihilation. For instance, in Fig. 1, terminals are branched from
junctions and they can be removed pairwisely by reducing the branching
distance. These defects, known as topological defects, are widely recognized
for playing a crucial role in the formation of periodic structures. By
detecting the positions of defects, the evolution of the labyrinthine patterns
would be systematically quantified. Specifically in the context of magnetic
labyrinthine patterns, a much smaller dataset was used to detect these defects
manually [4]. Recently, persistent homology has been used to extract the
topological features of the labyrinthine patterns in a systematic way [38].
However, it is still difficult to detect the positions of defects and classify
them into junctions and terminals.

In this article, we achieved the systematic detection. To quantify the
evolution of the labyrinthine patterns, we applied the TM-CNN technique to the
444 images and investigated how the number of defects and their locations
changes step by step. Fig. IV-B illustrates the step dependence of the number
of junctions and terminals during the demagnetization process. Results from 6
different experimental runs were averaged to estimate the number of defects.
In the initial stages of the demagnetization process, there are approximately
800 defects. The number of defects sharply decreases around step 5 and remains
nearly unchanged from step 10 onward. Consequently, the transition to the
quenched state is inferred to start before step 5 and be completed by step 10.
The reduction in the number of defects during the demagnetization process is
compatible with the naive expectation based on the physical argument of a
decrease in the number of defects associated with the increased spatial
coherence of the stripes. Furthermore, the number of junctions and terminals
are close, which is consistent with the topological argument that junctions
and terminals are paired.

Further quantification of the evolution of the labyrinthine patterns based on
the spatial distribution of the defects (and not only on their number) is
discussed in another article [6].

[h](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.45]okubo25.png Experimental
averages of junction and terminal counts per step with one standard deviation
shaded.

##  V Conclusion

In this work, we presented a new algorithm named TM-CNN to detect defects in
magnetic labyrinthine patterns, contributing to a pioneering analysis in
material science. Our study characterized the evolution of junctions and
terminals in magnetic stripes during demagnetization procedures, aiming at
better understanding defect arrangement in magnetic materials [6].

TM-CNN employs a two-stage detection procedure, combining template matching
for initial detection and a convolutional network classifier for refining
misdetections. This approach ensures a high detection accuracy and facilitates
dataset annotation through a semi-automatic procedure.

In our experiments, TM-CNN exhibited performance superior to other techniques,
achieving an impressive F1 score of 0.988. This high performance is mainly due
to TM-CNN’s ability to locate small and clustered objects. TM-CNN achieves
almost 100% accuracy with a simple CNN classifier with less than half a
million parameters and can be used even on computers without GPUs.

While TM-CNN was developed for defect detection in labyrinthine magnetic
patterns, its potential applications are not limited to this field. Future
research could explore the use of TM-CNN in other domains, such as identifying
bifurcations in blood vessels or adapting it to other structures that can be
modeled using templates.

## References

  * [1] Masaya Uchida, Yoshinori Onose, Yoshio Matsui, and Yoshinori Tokura.  Real-space observation of helical spin order.  Science, 311(5759):359–361, 2006.
  * [2] T. Garel and S. Doniach.  Phase transitions with spontaneous modulation-the dipolar ising ferromagnet.  Phys. Rev. B, 26:325–329, Jul 1982.
  * [3] P. Molho, J. L. Porteseil, Y. Souche, J. Gouzerh, and J. C. S. Levy.  Irreversible evolution in the topology of magnetic domains (invited).  Journal of Applied Physics, 61(8):4188–4193, 04 1987.
  * [4] M. Seul, L. R. Monar, L. O’Gorman, and R. Wolfe.  Morphology and local structure in labyrinthine stripe domain phase.  Science, 254(5038):1616–1618, 1991.
  * [5] Naoto Nagaosa and Yoshinori Tokura.  Topological properties and dynamics of magnetic skyrmions.  Nature Nanotechnology, 8(12):899–911, Dec 2013.
  * [6] Kotaro Shimizu, Vinicius Yu Okubo, Rose Knight, Ziyuan Wang, Joseph Burton, Hae Yong Kim, Gia-Wei Chern, and B. S. Shivaram.  Machine Learning Assisted Characterization of Labyrinthine Pattern Transitions.  arXiv:2311.10558, 2023.
  * [7] Kai Briechle and Uwe D. Hanebeck.  Template matching using fast normalized cross correlation.  In David P. Casasent and Tien-Hsin Chao, editors, Optical Pattern Recognition XII, volume 4387, pages 95 – 102. International Society for Optics and Photonics, SPIE, 2001.
  * [8] P. Viola and M. Jones.  Rapid object detection using a boosted cascade of simple features.  In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pages I–I, 2001.
  * [9] N. Dalal and B. Triggs.  Histograms of oriented gradients for human detection.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 886–893 vol. 1, 2005.
  * [10] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  25, 2012.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation, 2014.
  * [13] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision, 88(2):303–338, June 2010.
  * [14] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks, 2016.
  * [15] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection, 2016.
  * [16] Yang Liu, Peng Sun, Nickolas Wergeles, and Yi Shang.  A survey and performance evaluation of deep learning methods for small object detection.  Expert Systems with Applications, 172:114602, 2021.
  * [17] Ricardo H Maruta, Hae Yong Kim, Danilo R Huanca, and Walter J Salcedo.  A new correlation-based granulometry algorithm with application in characterizing porous silicon nanomaterials.  ECS Transactions, 31(1):273, 2010.
  * [18] Hae Yong Kim, Ricardo Hitoshi Maruta, Danilo Roque Huanca, and Walter Jaimes Salcedo.  Correlation-based multi-shape granulometry with application in porous silicon nanomaterial characterization.  Journal of Porous Materials, 20:375–385, 2013.
  * [19] Sidnei Alves De Araújo, Jorge Henrique Pessota, and Hae Yong Kim.  Beans quality inspection using correlation-based granulometry.  Engineering Applications of Artificial Intelligence, 40:84–94, 2015.
  * [20] Gui-Song Xia, Julie Delon, and Yann Gousseau.  Accurate junction detection and characterization in natural images.  International Journal of Computer Vision, 106(1):31–56, Jan 2014.
  * [21] Harry Pratt, Bryan M. Williams, Jae Yee Ku, Charles Vas, Emma McCann, Baidaa Al-Bander, Yitian Zhao, Frans Coenen, and Yalin Zheng.  Automatic detection and distinction of retinal vessel bifurcations and crossings in colour fundus photography.  Journal of Imaging, 4(1), 2018.
  * [22] Sheng He, Marco Wiering, and Lambert Schomaker.  Junction detection in handwritten documents and its application to writer identification.  Pattern Recognition, 48(12):4036–4048, 2015.
  * [23] M Elena Martinez-Perez, Alun D Hughes, Alice V Stanton, Simon A Thom, Neil Chapman, Anil A Bharath, and Kim H Parker.  Retinal vascular tree morphology: a semi-automatic quantification.  IEEE Trans Biomed Eng, 49(8):912–917, August 2002.
  * [24] K. Liu, Y.S. Huang, and C.Y. Suen.  Identification of fork points on the skeletons of handwritten chinese characters.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10):1095–1100, 1999.
  * [25] Chungnan Lee and Bohom Wu.  A chinese-character-stroke-extraction algorithm based on contour information.  Pattern Recognition, 31(6):651–663, 1998.
  * [26] Michael Maire, Pablo Arbelaez, Charless Fowlkes, and Jitendra Malik.  Using contours to detect and localize junctions in natural images.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8, 2008.
  * [27] Ran Su, Changming Sun, and Tuan D. Pham.  Junction detection for linear structures based on hessian, correlation and shape information.  Pattern Recognition, 45(10):3695–3706, 2012.
  * [28] R. Deriche and T. Blaszka.  Recovering and characterizing image features using an efficient model based approach.  In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 530–535, 1993.
  * [29] He Zhao, Yun Sun, and Huiqi Li.  Retinal vascular junction detection and classification via deep neural networks.  Computer Methods and Programs in Biomedicine, 183:105096, 2020.
  * [30] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick.  Mask R-CNN.  CoRR, abs/1703.06870, 2017.
  * [31] I B Puchalska, G A Jones, and H Jouve.  A new aspect on the observation of domain structure in garnet epilayers.  J. Phys. D: Appl. Phys., 11(15):L175, oct 1978.
  * [32] Hae Yong Kim and Sidnei Alves de Araújo.  Grayscale template-matching invariant to rotation, scale, translation, brightness and contrast.  In Domingo Mery and Luis Rueda, editors, Advances in Image and Video Technology, pages 100–113, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg.
  * [33] J. P. Lewis.  Fast normalized cross‐correlation.  Vision Interface, 95:120, 1995.
  * [34] Sidnei Alves de Araújo and Hae Yong Kim.  Ciratefi: An rst-invariant template matching with extension to color images.  Integr. Comput.-Aided Eng., 18(1):75–90, jan 2011.
  * [35] Hae Yong Kim.  Rotation-discriminating template matching based on fourier coefficients of radial projections with robustness to scaling and partial occlusion.  Pattern Recognition, 43(3):859–872, 2010.
  * [36] Li Deng.  The mnist database of handwritten digit images for machine learning research [best of the web].  IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  * [37] Christian Eggert, Stephan Brehm, Anton Winschel, Dan Zecha, and Rainer Lienhart.  A closer look: Small object detection in faster r-cnn.  In 2017 IEEE International Conference on Multimedia and Expo (ICME), pages 421–426, 2017.
  * [38] T. Yamada, Y. Suzuki, C. Mitsumata, K. Ono, T. Ueno, I. Obayashi, Y. Hiraoka, and M. Kotsugi.  Visualization of topological defect in labyrinth magnetic domain by using persistent homology.  Vacuum and Surface Science, 62(3):153–160, 3 2019.

|  Vinícius Yu Okubo was born in Campinas, Brazil, in 1999. He completed his
B.S. in electrical engineering from the University of São Paulo in 2022.
Currently, he is pursuing his M.S. in electrical engineering at the University
of São Paulo. His academic and research interests are focused on computer
vision and deep learning.
---|---
|  Kotaro Shimizu was born in Tokyo, Japan, in 1996. He received his B.S.
degree in Physics from Waseda University, Japan, in 2019 and M.S. degree in
Physics from the University of Tokyo, Japan, in 2021. He has been pursuing his
Ph.D. in Physics as a JSPS research fellowship for young scientists in the
University of Tokyo since 2021. He is a theoretical physicist working in
condensed matter physics. His main research interest lies in magnetism,
strongly correlated electrons, and topological phases of matter.
---|---
|  B. S. Shivaram (Ph.D.’84) received his B.S. degree in Physics, Chemistry
and Mathematics from Bangalore University, India, in 1977 and the M.S. degree
in Physics from the Indian Institute of Technology, Madras, India, in 1979 and
his Ph.D. in experimental condensed matter physics from Northwestern
University, Evanston, Illinois in 1984. From 1985 to 1986, he was a Research
Associate at the James Frank Institute of University of Chicago, USA. Since
1987 he has been on the faculty in the department of physics at the University
of Virginia, Charlottesville, Virginia, USA. He is an experimental physicist
working in a broad area of condensed matter including magnetism. He is the
recipient of the Alfred P. Sloan foundation fellowship and serves currently as
the Vice-Chair of the Forum for Outreach and Engaging the Public of the
American Physical Society.
---|---
|  Hae Yong Kim was born in Korea. He received the B.S. and M.S. degrees (with
distinctions) in computer science and the Ph.D. degree in electrical
engineering from the Universidade de São Paulo (USP), Brazil, in 1988, 1992
and 1997, respectively. He is currently an Associate Professor with the
Department of Electronic Systems Engineering, USP. He is the author of more
than 100 articles and holds three patents. His research interests include
image processing, machine learning, medical image processing and computer
security. Dr. Kim and colleagues received the 6th edition of the Petrobras
Technology Award in the “Refining and Petrochemical Technology” category
(2013); the “Best Paper in Image Analysis” award at the Pacific-Rim Symposium
on Image and Video Technology (2007); and the Thomson ISI Essential Science
Indicators “Hot Paper” award, for writing one of the top 0.1% of the most
cited computer science papers (2005).
---|---


11institutetext: Nexa Center for Internet & Society, DAUIN, Politecnico di
Torino, Italy
11email<EMAIL_ADDRESS>22institutetext: Future Urban Legacy Lab, Politecnico di Torino, Italy
33institutetext: Fondazione Bruno Kessler, Italy
33email<EMAIL_ADDRESS>
# The invisible power of fairness. How machine learning shapes democracy

Elena Beretta and Antonio Santangelo and Bruno Lepri and Antonio Vetrò and
Juan Carlos De Martin 1133 0000-0002-6090-2765 11 0000-0001-8885-6214 33
0000-0003-1275-2333 1122 0000-0003-2027-3308 11 0000-0002-7867-1926

###### Abstract

Many machine learning systems make extensive use of large amounts of data
regarding human behaviors. Several researchers have found various
discriminatory practices related to the use of human-related machine learning
systems, for example in the field of criminal justice, credit scoring and
advertising. Fair machine learning is therefore emerging as a new field of
study to mitigate biases that are inadvertently incorporated into algorithms.
Data scientists and computer engineers are making various efforts to provide
definitions of fairness. In this paper, we provide an overview of the most
widespread definitions of fairness in the field of machine learning, arguing
that the ideas highlighting each formalization are closely related to
different ideas of justice and to different interpretations of democracy
embedded in our culture. This work intends to analyze the definitions of
fairness that have been proposed to date to interpret the underlying criteria
and to relate them to different ideas of democracy.

###### Keywords:

machine learning fairness equity discrimination.

##  1 Introduction

Nowadays, machine learning systems make an extensive use of large amounts of
data on human behaviors, collected through various channels (e.g. social
media, apps, mobile phone usage, credit card transactions, etc.). The
widespread resort to these data for disparate purposes is rapidly transforming
several domains of our daily life. However, the increasing use of automated-
software and machine learning systems, for example in the field of criminal
justice [2], advertising [25], and credit scoring [15], is raising a wide
range of legal and ethical issues. It is obviously impossible to provide an
exhaustive mapping of all the problems, but through an easy expedient a useful
logical scheme summarizing the various ones can be supplied. As a matter of
fact, if we move from one of the many elementary definitions of “algorithm” -
an encoded procedure to transform input (data) in output (expected result)
through a series of calculations - we understand that we are confronted with
three constitutive elements: 1) input; 2) procedure; 3) output. A major
concern in the first phase is the presence of biases in the input dataset
(this is not the only one: for example, personal data protection is also a
relevant issue). In the second phase, a largely debated issue is the
transparency and accessibility of the procedure - the problem of the black box
[21]; while the main problematic aspect of the third phase is the possible
discriminatory effects of the algorithmic decision. A given ethical and/or
legal problem must be studied with reference to: a) the phase in which it is
mainly located, and b) the possible propagation of the issue throughout the
algorithm’s elaboration path. Since many application domains of machine
learning algorithms are not protected by law against discrimination, the
attention of actors belonging to different sectors is increasingly focusing on
the way algorithms encode prejudices and lead to disproportionate results
[20]. As a further development of the emergence of adverse outcomes, many
researchers are involved in finding solutions to overcome the problem of
discrimination in automated software systems by embedding the idea of fairness
in the algorithm’s structure. In this paper, we contribute to the debate on
fairness in machine learning by discussing the impact that its recent
mathematical formalizations may have on societies. In particular, we reflect
on the meaning of each definition of fairness in relation to the different
ideas of democracy and equality they take with them. Our hypothesis is
twofold: on one side, we suppose different democratic cultures may foster the
diffusion of different definitions of fairness; on the other side, the
different ways of designing machine learning tools fit in different ways in
the various societies and may fit better than others, especially because in a
same social contexts various ideas of democracy may coexist. We describe in
Section 2 the role of fairness in the field of machine learning; several
recent mathematical implementations of fairness and various democracy
typologies are exposed and analyzed in Subsection 2.1 and Subsection 2.2,
respectively. In Section 3 we discuss commonalities among fairness definitions
and democracy typologies, pointing out challenges and open issues for the fair
machine learning research domain. Finally, Section 4 draws some conclusions
and discuss potential future works on this topic.

##  2 Why fairness matters?

The way machine learning systems act is a crucial matter for our societies
[20]. Algorithms are designed to recognize situations leading to satisfactory
outcomes, they are modeled to look for patterns and characteristics in
individuals that have historically brought to success, thereby not making
things fair if randomly employed, but replicating models and past practices. A
relevant example is the recent case, exposed by the investigative website
ProPublica111https://www.propublica.org/article/machine-bias-risk-assessments-
in-criminal-sentencing, about the COMPAS recidivism tool, an algorithm used to
inform criminal sentencing decisions by predicting recidivism. In particular,
the study found that COMPAS is evaluating useful risk factors for
classification, such as socio-economic status and type of crime, which share
high covariance with some sensitive attributes, in this case the race.
Although the reason for this high covariance may be the result of a
preexisting prejudice present in a wider system than one considered for the
analysis of recidivism , it should be underlined that the study found that
COMPAS was significantly more likely to falsely label as high-risk of
recidivism black defendants because of the above mentioned interaction, and to
underestimate the risk of white defendants by less accurately detecting the
false negative rate. Although research is increasingly concentrating in
finding solutions to mitigate the segregating effect of some algorithms, many
efforts will have to be done to consider when an algorithm fails, for whom it
fails and what are the social costs of the failure [9]. Hence, designing a
fair algorithm entails two aspects, closely related to each other: firstly, to
evaluate the meaning of choosing one kind of fairness instead of another in a
certain society; and secondly, to assess the degree of social acceptability
subordinated to the context and to the selected fairness criterion.
Considering that more than one definition of fairness cannot be achieved
simultaneously [18], design choices have a relevant impact on the effect that
the algorithm’s outcomes will have on society. It is therefore quite relevant
starting to figure out which kind of societal values and democratic concepts
are tied to the current mathematical formalizations of fairness. In the
following sections, we analyze the most relevant mathematical definitions of
fairness provided in the machine learning research area, assuming that a
decision on how to design a fair machine learning system could be acceptable
with more probability as much as it corresponds to the ideas of democracy
shaped by individuals in the social context in which it will be used.

###  2.1 Fairness in machine learning domain

In the recent years, several formal definitions of fairness have been
suggested by the machine learning community. In Table 1, we report the most
widespread ones grouped by similar characteristics: in particular, the first
column indicates the name of the partitioning, while the second one the
extended name of the fairness definition. Finally, the third column contains
scientific references which are then further specified in Table 2.

Partition | Definition | Reference
---|---|---
Group | Statistical | [1, 3, 9]
fairness | parity |
| Accuracy | [10]
| parity |
| False positive | [5, 7]
| parity |
| Positive rate | [4, 11, 12]
| parity |
| Predictive parity | [6, 7]
| Predictive value parity | [3]
| Equal | [4, 7, 9]
| opportunity |
| Equal | [4, 7]
| threshold |
| Well-calibration | [2]
| Balance for | [2]
| positive class |
| Balance for | [2]
| negative class |
Individual |  | [1]
fairness |  |
Counterfactual |  | [9]
fairness |  |
Preference-based | Preferred treatment | [8]
fairness | Preferred impact | [8]
Fairness through |  | [1, 4, 9]
unawareness |  |
Table 1: Fairness in machine learning literature Reference | Paper
---|---
| number
[12] | [1]
[18] | [2]
[2] | [3]
[15] | [4]
[9] | [5]
[24] | [6]
[7] | [7]
[28] | [8]
[19] | [9]
[10] | [10]
[27] | [11]
[4] | [12]
Table 2: References

First of all, we provide some mathematical notations that compose a typical
setup in a machine learning domain: X denotes the features of an individual; Y
denotes the target variable; A denotes a sensitive attribute (i.e. gender,
race, etc.); C denotes a classifier; S denotes a score function or a
conditional expectation. For example, the frequency of an event given certain
observed characteristics can be written as S = E[Y$\mid$X]; t is a threshold.
In case of binary classifiers, the score value causes the acceptance of
classifier outputs when it is above t, otherwise causes the rejection.

Then, we introduce and briefly describe the fairness definitions listed in
Table 1, supplied with examples regarding risk assessment in the criminal
justice domain. Individuals rated high risk of re-offending are classified by
0, otherwise 1 - that means low risk of recidivism. The variable race has been
considered as a sensitive and protected attribute.

#### Group fairness.

Below, we introduce several formal definitions of group fairness.

Statistical parity. Classifier C satisfies statistical parity if Pa(C = 1) =
Pb(C = 1) for all groups a, b \- i.e. a = black, b = white. This means that
both black and white people should have equal probability to be classified as
low risk.

Accuracy parity. Classifier C satisfies accuracy parity if Pa(C = Y) = Pb(C =
Y) for all groups a, b. This means that both black and white people should
have equal probability to be correctly classified as low risk, if belonging to
actual low risk rate, and correctly classified as high risk, if belonging to
actual high risk rate.

False positive parity. Classifier C satisfies false positive parity if Pa(C =
1$\mid$Y = 0) = Pb(C = 1$\mid$Y = 0) for all groups a, b. This means that both
black and white people with actual high risk rate should have equal
probability to be incorrectly classified as low risk (False Positive Rate).

Positive rate parity. Classifier C satisfies positive rate parity if Pa(C =
1$\mid$Y = i) = Pb(C = 1$\mid$Y = i), i $\in$ 0, 1, for all groups a, b. This
means that both black and white people should have equal probability to be
incorrectly classified as low risk - False Positive Rate - and to be correctly
classified as low risk (True Positive Rate).

Predictive parity. Classifier C satisfies predictive parity if Pa(Y = 1$\mid$C
= 1) = Pb(Y = 1$\mid$C = 1), for all groups a, b. This means that both black
and white people with low risk predicted score (Positive Predictive Value)
should have equal probability to really belong to the low risk class.

Predictive value parity. Classifier C satisfies predictive value parity if
(Pa(Y = 1$\mid$C = 1) = Pb(Y = 1$\mid$C = 1)) $\land$ (Pa(Y = 0$\mid$C = 0) =
Pb(Y = 0$\mid$C = 0)) for all groups a, b. This means that both black and
white people with low risk predicted score (Positive Predicted Value) should
have equal probability to really belong to low risk class, and both black and
white people with high risk predicted score (Negative Predictive Value) should
have equal probability to really belong to high risk class.

Equal opportunity. Classifier C satisfies equal opportunity if Pa(C = 1$\mid$Y
= 1) = Pb(C = 1$\mid$Y = 1) for all groups a, b. This means that both black
and white people with actual low risk rate should have equal probability to be
incorrectly classified as high risk (False Negative Rate). Since
mathematically a classifier that satisfies False Negative Rate equity
satisfies at the same time True Positive Rate equity, the definition also
implies that both black and white people with actual low risk rate should have
equal probability to be correctly classified as low risk.

Equal threshold. Classifier C satisfies equal threshold if Pa(Y = 1$\mid$S =
s) = Pb(Y = 1$\mid$S = s), s $\in$ [0, 1], for all groups a, b. This means
that both black and white people should have equal score threshold t under
which they are classified at low risk, and above which they are classified at
high risk.

Well-calibration. Classifier C satisfies well-calibration if Pa(Y = 1$\mid$S =
s) = Pb(Y = 1$\mid$S = s) = s, s $\in$ [0, 1], for all groups a, b. This means
that both black and white people with the same score should be treated
comparably “with respect to the outcome, rather than treating black and white
people with the same score differently based on the race group they belong
to”[18].

Balance for positive class. Classifier C satisfies balance for positive class
if Ea(S$\mid$Y = 1) = Eb(S$\mid$Y = 1), for all groups a, b. This means that
both black and white people with an actual low risk rate should have the same
expected value assigned by the classifier C (a classifier uses the
characteristics of individuals to identify which class - or group - they
belong to). That is to say, it should not happen that the scoring process is
“systematically more inaccurate for negative cases - high risk score - in one
group than the other”[18].

Balance for negative class. Classifier C satisfies balance for negative class
if Ea(S$\mid$Y = 0) = Eb(S$\mid$Y = 0), for all groups a, b. This means that
both black and white people with an actual high risk rate should have the same
expected value assigned by the classifier C. That is to say, it should not be
that the scoring process is ”systematically more inaccurate for positive cases
- low risk score - in one group than the other”[18].

#### Individual fairness.

Given a set of individuals V and a set of outcomes A = $\left\\{0,1\right\\}$,
and considering a metric on individuals d: V x V $\to$ R and randomized
mappings M: V $\to$ $\Delta$A, individuals fairness is achieved if a
randomized classifier, mapping individuals to distributions over outcomes,
minimizes expected loss subject to the (D,d)-Lipschitz condition of D(Mx, My)
$\leq$ d(x, Y) [12].
This means that two individuals are similarly classified if they are
considered similar with respect to a particular task, such as to pay off a
debt with a bank.

#### Counterfactual fairness.

Classifier C satisfies counterfactual fairness if P(CA←a (U222”U is a set of
latent background variables, which are factors not caused by any variable in
the set V of observable variables”[19]) = y$\mid$X = x, A = a) =
P(CA←${}_{a^{\prime}}$ (U) = y$\mid$X = x, A = a). That is, given a set of
attributes (education level, type of crime, drugs problems and protected
attribute A = race) and an outcome $\hat{Y}$ to be predicted (recidivism), a
graph is counterfactually fair if race is not directly linked to $\hat{Y}$
through any other attributes.

Intuitively, this means that a decision is fair towards an individual if it is
the same in (i) the actual world and (ii) a counterfactual world where the
individual belonged to a different demographic group (i.e. white instead of
black).

#### Preference-based fairness.

Here, we introduce new formalization of fairness [28] that are inspired by the
concepts of fair division in economics and game theory [26, 3].

Preferred treatment. Classifier C satisfies preferred treatment if Ba(Ca)
$\geq$ B${}_{a^{\prime}}$(C${}_{a^{\prime}}$), for all a, a’ $\in$ A333Ba is
the fraction of beneficial outcomes received by users sharing a certain value
of the sensitive attribute a[28]. This means that the preferred condition is
preserved if each group obtains more benefit from their own classifier then it
would be assigned from any other classifier. In other words, both black and
white people should prefer “the set of decisions they receive over the set of
decisions they would have received had they collectively presented themselves
to the system as members of a different sensitive group”[28].

Preferred impact. Classifier C satisfies preferred impact if Ba(C) $\geq$
Ba(C’), for all a $\in$ A. This means that the preferred condition is
preserved if a classifier C, with respect to any other classifier, assigns at
least the same benefit for all groups. In other words, both black and white
people should prefer “the set of decisions they receive over the set of
decisions they would have received under the criterion of impact parity”[28].

#### Fairness through unawareness.

Classifier C satisfies fairness through unawareness if X: Xa = Xb $\to$ Ca =
Cb for both individuals a, b. This means that for example the attribute race
should not be used to train the classifier and thus to take a decision (i.e.
granting or not a loan).

###  2.2 Fairness and democracy

In order to understand which idea of fairness can be affirmed in a certain
society, it may be useful to relate it to the type of democracy in force
within the latter. In this way, we may reflect on the concept of equality that
this idea of democratic life brings with it. As the philosopher Christiano
writes [8], democracy is a procedure for taking collective decisions
characterized by the fundamental equality of those who participate in it.
According to Bozdag and Van Den Hoven [6], there are at least four different
ways of conceiving democracy, which must be seen as some sort of ideal types,
that interact and confront each other in reality, giving rise to the different
democratic regimes in force in the world. The first is the model of liberal
democracy, commonly centred on the defence of fundamental rights and freedoms
(i.e. self-determination, choice, private property, expression, etc.) from all
forms of coercive power [11], [16]. It is an individualistic vision of
democratic life, according to which democracy must be the place that preserves
the freedoms of the individual, characterizing itself as a procedure which,
through electoral competition and voting, represents and aggregates the will
of individuals. In this case, the criterion of justice on which the concept of
equality between persons is based denies the maxim of egalitarianism,
according to which all men must be (at the very least) equal in everything,
but admits the equality of all only in something, that is, in the so-called
fundamental rights [5, p. 95]. The second model is that of deliberative
democracy [6], seen as a way of taking collective decisions, based on a
confrontation between free and equal citizens. This confrontation must be
aimed at finding logical solutions to satisfy the good of all. Here, the
emphasis is less on the individual and more on her/his participation in common
life. Democracy is not seen as the mere aggregation of the will of individuals
in a vote, but as a place where people can talk to each other, make their
points of view count and try to understand, together, which is the best one.
The fulcrum of all this is the exercise of a rationality, in a certain sense,
devoid of the will to affirm partisan interests, which is very similar to the
rationality theorized by Habermas [14] and Rawls [22], [23]. In this case,
equality is above all in the possibility of having access to the deliberative
process and in ensuring that one can confront others, even if s/he is part of
a minority. The third model of democracy is what Bozdag and Van Den Hoven [6]
call republican or protesting. It is based on the idea that all citizens must
be free from the arbitrary exercise of power by someone. For this reason, they
must be able to control the activity of those who govern them, being at the
same time able to challenge it. More than the consensus of deliberative
democracy, therefore, in this case it is fundamental the power to challenge
choices and points of view that are not shared. It is very important the
transparency and publicity of the actions of the rulers, and it is essential
that anyone can question them, in a way that can produce some effect. This is
the kind of equality that the supporters of this model of democracy are aiming
for. The fourth model is that of competitive democracy [17]. It logically
contrasts with that of deliberative democracy, because its proponents do not
believe that people, by confronting and using pure rationality, can reach an
agreement on the common good. Often, the positions of each one are based on
passions, partisan interests, instances that one does not want or cannot
negotiate. Democracy, then, must consist of a set of procedures that allow the
supporters of the various positions to express themselves and to confront each
other, in order to prevail. This must be done knowing that the opponents,
having lost a confrontation, will have the opportunity to assert their demands
in future occasions. Equality, in this case, is similar to the one existing in
a competitive context, in which the participants have, at the beginning, the
same opportunities to assert themselves. It is defined, in fact, as equality
of opportunity [5, p. 24-26]. To these four models, however, it is necessary
to add a fifth that we could define as egalitarian democracy, following
Bobbio’s reasoning [5, p. 26-38]. It is based on the ideal of guaranteeing a
de facto equality which, as Bobbio himself writes (ibidem, p. 27), is economic
equality. In practice, egalitarian democracy tries to reduce the distance
between those who have less and those who have more, with equalizing
mechanisms such as progressive taxes, subsidies to the poor, and so on. Hence,
the principle on which it is based is the opposite of that of competitive
democracy, in the sense that its proponents recognize the differences between
people rather than what they have in common, and try to favor those who are
disadvantaged.

##  3 Results

In the previous section we exposed the most widespread fairness criteria
formalized in the machine learning domain, observing that the achievement of
fairness is subordinated to various probabilistic constraints, each of them
related to one or more parts which an algorithm is composed of (see Section
1). The connection among machine learning approaches and democratic structures
similarly exists: as well as fair parameters can be encoded both in procedure
and in algorithms’ outcome, even democratic values can be accomplished both in
procedural and in outcome part of a decision. Therefore, as we show in Figure
1, selecting a fairness criterion instead of another is a trade-off process
where the decision-maker is called upon to determine if the achievement of the
desired goal entails preserving parity (all individuals are equal on the basis
of some protected attributes) or satisfying preferences (individuals may have
different preferences independent from protected attributes), and if it has to
be reached by means of procedure constraints or impact factors. In other
words, should the algorithm be fair in the way it takes a decision or should
the decision be fair itself? The first column refers to the practice of
preserving a kind of equality among people, while the second one refers to the
practice of satisfying preferences; rows indicate in which part of an
algorithm the fairness criteria are embedded and if inequality is mitigated
from the beginning or in a second phase. The following partitioning (Figure
1), inspired by Gajane and Pechenizkiy [13] is the result of our reasoning,
which is explained in details for each democracy type in the following
subsections.

Figure 1: Trade-off in fairness selection process related to democracy
typologies. Legend: Competitive (green), Liberal (yellow), Egalitarian
(orange)

#### Liberal democracy

Liberal democracy is a model based on self-determination and preservation of
individual freedom. Analyzing the mathematical formulation of individual
fairness, we believe that such a notion of fairness underlies an idea of
liberal democracy; in fact, even though attention is paid to protect
minorities’ rights, individual diversity is chiefly preserved in preferential
way, considering individuals not because of an intrinsic differentiation but
rather because of some different tasks for which they are similarly treated.

#### Competitive democracy

This kind of democracy expresses the idea that actors cannot reach an
agreement among themselves on a specific result by reason of utility; this
reasoning implies the condition occurring when allocation of goods is such
that it is not possible to improve the condition of one actor without
worsening the condition of another one. So the best that can be done is
guaranteeing equality of opportunity among groups through preserving and
protecting minorities’ rights. Therefore, from the point of view of
competitive democracy, the mechanism of satisfying preferences could be seen
as considering like a game which each individual should have the same
opportunity to play. In this sense, almost all the fairness definitions that
we have reported reflect the competitive democracy criteria; in fact, the
majority of them encodes the idea that all people should have the same
possibility of being correctly classified (i.e. each individual should have
the same opportunity to receive a loan if s/he deserves it). From a
statistically point of view, fairness definitions clustered in the group
fairness partition are not substantially very different, the disparity mainly
consists on the observer’s point of view, or rather the stakeholder’s; i.e.,
in statistical parity we assume the society’s point of view that wonder ”Is
the selected set demographically balanced?”; in false positive parity we
assume the defendant’s point of view that wonder ”What is the probability that
I’m incorrectly classified as high-risk?”; in predictive value parity we
assume the decision-maker point of view that wonder ”Of those I have labeled
at high risk, how many will be recidivists?”.

#### Egalitarian democracy

This type of democracy is inspired to Bobbio’s notion of distributive
egalitarianism, intended as a way of redistributing material resources between
advantaged and disadvantaged people [5, p. 30-38]. But if we extend the idea
of redistribution also to political resources, knowledge, etc., then we can
apply it to the fairness criteria we have reported. In fact, no definition of
fairness completely fulfills egalitarian democracy criteria, but preference-
based models are the closest ones; in fact, both definitions of preference-
fairness refer to the concept of benefit, which as expressed in the definition
of Zafar _et al._ [28] is very similar to the concept of favoring individuals
belonging to minorities. We exclude republican and deliberative democracy from
the general classification reported in Figure 1. The problems that are faced,
when reflecting on the republican model of democracy, are usually touched,
within the ethics of data and algorithms, when referring to the issues of
transparency and accountability of computer tools. Since the attention mainly
focuses on monitoring who exercises the power and on controlling the
decisions, we believe actually the whole general ground of fair machine
learning has been inspired by republican democracy values. But none of the
definitions of fairness we have reported have directly to do with transparency
and accountability, which have to be considered as the results of the whole
fairness process.

Instead, deliberative democracy is a model based on agreement and on the idea
that all individuals are free and equal to each others, when having the
possibility of seeing their instances represented and taken into consideration
in the contexts where decisions about them and their lives are taken. Its
ethical premises are very similar to the ones of competitive democracy. In
fact, strong attention is given to preserve minorities’ rights and to give
everyone equal possibilities to participate to the game of democratic debate.
But once again, we think none of the fairness definitions we have reported has
directly to do with guaranteeing that democratic decisions are taken
acknowledging the instances of everybody. Moreover, we believe at the present
the field of fair machine learning still lacks fairness definitions built on
egalitarian democratic principles. In fact, if almost all fairness definitions
are to some extent based on the principle that everyone should have equality
of opportunities, egalitarian democratic definitions can be considered in
contrast to this principle, because they rely on a sort of an inequality
principle, according to which a disadvantaged individual must be favored over
others. Criteria relating to most of the fairness definitions are linked to
competitive democracy which may be a following stage of the egalitarian
democracy, because competitive democracy rules, according to which everyone
should have the opportunity to fairly participate at the competition, can be
better implemented if inequalities are previously resolved through a
distributive logic.

##  4 Discussion and Conclusions

Over the last decade machine learning has radically changed the way we take a
decision. Many researchers studied the performances of statistical models and
human judgments, demonstrating how in some circumstances models based on
machine learning have surpassed those based on human judgments ([1]).
Moreover, by now machine learning algorithms are well known to use a large-
amount of data to induce a particular rule starting from a generalized datum;
in this article we have addressed problems related to the correctness of these
rules. The examples enhancing machine learning models have in fact shown how
often the results reflect bias and human prejudices in different contexts -
especially those in which we try to reproduce and analyze human behavior.
Hence, several efforts have been devoted to finding solutions that re-
calibrate balances in outputs of machine learning systems. However, the debate
around fairness the in machine learning domain displays a profound and
relatively worrying lack: although researchers are acting and reacting in a
positive and proactive way, data scientists and computer engineers are
increasingly involved in taking decisions that affect individuals, operating
as judges in decision-making processes and constituting a sort of invisible
power \- although in a positive way - whereas society at large should play
this role. In this article, we enrich these efforts and the related debate,
highlighting that meanings of fairness, underlying to statistical constraints,
and the democratic values are strictly connected; in fact, we carry on the
idea that the spread of one fairness definition instead of another is highly
justified by the ideas of justice and democracy shared among the society in
which the above-mentioned fairness criterion is selected. We consider this
qualitative analysis as a prelude of quantitative researches. For example, our
study may pave the way for the implementation of an agent-based model that
mimics the decision-making, where agents are informed by different kind of
democratic values. The aim of the model would be to analyze which statistical
parameters lead to specific fairness criteria and under which probabilities.
Moreover, we plan to further develop our work by proposing a new formalization
of fairness based on egalitarian democratic principles.

## References

  * [1] Barocas, S., Hardt, M., Narayanan, A., Fairness and Machine Learning. fairml-book.org (2018), http://www.fairmlbook.org
  * [2] Berk, R., Heidari, H., Jabbari, S., Kearns, M., Roth, A., Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research (2018)
  * [3] Berliant, M., Thomson, W., On the fair division of a heterogeneous commodity. Journal of Mathematics Economics (1992)
  * [4] Binns, R., Fairness in machine learning: Lessons from political philosophy. In:Proceedings of Machine Learning Research. vol. 81, pp. 149–159. Sorelle A. Friedler, Christo Wilsonf (2018)
  * [5] Bobbio, N., Eguaglianza e libertá. Einaudi, Torino, Italy (1995)
  * [6] Bozdag, E., van den Hoven, J., Breaking the filter bubble: democracy and design.Ethics and Information Technology17, 249–265 (2015)
  * [7] Chouldechova, A., Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data (2017)
  * [8] Christiano, T., Democracy. Stanford Encyclopedia of Philosophy (2006)
  * [9] Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., Huq, A., Algorithmic decision making and the cost of fairness. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2017)(2017)
  * [10] Dieterich, W., Mendoza, C., Brennan, T., Compas risk scales: Demonstrating ac-curacy equity and predictive parity. Tech. rep., Northpointe Inc. (2016)
  * [11] Dunn, J., Western political theory in the face of the future. Vol. 3. Cambridge University Press, Cambridge, UK (1979)
  * [12] Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemeln, R., Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference. pp. 214–226. ACM (2012)
  * [13] Gajane, P., Pechenizkiy, M., On formalizing fairness in prediction with machine learning. arXiv:1710.03184 (2018)
  * [14] Habermas, J., Between facts and norms: Contributions to a discourse theory of law and democracy. MIT Press, Cambridge, US (1998)
  * [15] Hardt, M., Price, E., Srebro, N., Equality of opportunity in supervised learning. In: Advances in Neural Information Processing Systems (2016)
  * [16] Held, D., Models of democracy. Stanford University Press, Palo Alto, US (2006)
  * [17] Held, D., The democratic paradox. Verso, London, UK (2009)
  * [18] Kleinberg, J., Mullainathan, S., Raghavan, M., Inherent trade-offs in the fair determination of risk scores. In: Proceedings of Innovations in Theoretical Computer Science (ITCS 2017) (2017)
  * [19] Kusner, M.J., Loftus, J.R., Russell, C., Silva, R., Counterfactual fairness. In: Proceedings of 31st Neural Information Processing Systems (NIPS 2017) (2017)
  * [20] O’Neil, C., Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown Publishing Group, New York, NY (2016)
  * [21] Pasquale, F., The Black Box Society: The Secret Algorithms That Control Money and Information. Harvard University Press, Cambridge, MA, USA (2015)
  * [22] Rawls, J., A theory of justice. Harvard University Press, Harvard, US (1971)
  * [23] Rawls, J., The idea of public reason. The MIT Press, Cambridge, US (1997)
  * [24] Simoiu, C., Corbett-Davies, S., Goel, S., The problem of infra-marginality in out-come tests for discrimination. The Annals of Applied Statistics (2017)
  * [25] Sweeney, L., Discrimination in online ad delivery. Queue - Storage11(3) (2013)
  * [26] Varian, H.: Equity, envy, and efficiency. Journal of Economic Theory (1974)
  * [27] Zafar, M.B., Valera, I., Rodriguez, M.G., Gummadi, K.P., Fairness beyond disparate treatment and disparate impact: Learning classification without disparate mistreatment. In: Proceedings of the 26th International Conference on World Wide Web (WWW 2017) (2017)
  * [28] Zafar, M.B., Valera, I., Rodriguez, M.G., Gummadi, K.P., Weller, A.: From parity to preference-based notions of fairness in classification. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017) (2017)