2022

[1]Mariette Awad

1]Electrical and Computer Engineering Department, American University of
Beirut, Riad El-Solh, Beirut, 1107 2020, Lebanon

# On The Potential of The Fractal Geometry and The CNNs’ Ability to Encode it

Julia El Zini<EMAIL_ADDRESS>Bassel Musharrafieh<EMAIL_ADDRESS><EMAIL_ADDRESS>[

###### Abstract

The fractal dimension provides a statistical index of object complexity by
studying how the pattern changes with the measuring scale. Although useful in
several classification tasks, the fractal dimension is under-explored in deep
learning applications. In this work, we investigate the features that are
learned by deep models and we study whether these deep networks are able to
encode features as complex and high-level as the fractal dimensions.
Specifically, we conduct a correlation analysis experiment to show that deep
networks are not able to extract such a feature in none of their layers. We
combine our analytical study with a human evaluation to investigate the
differences between deep learning networks and models that operate on the
fractal feature solely. Moreover, we show the effectiveness of fractal
features in applications where the object structure is crucial for the
classification task. We empirically show that training a shallow network on
fractal features achieves performance comparable, even superior in specific
cases, to that of deep networks trained on raw data while requiring less
computational resources. Fractals improved the accuracy of the classification
by 30% on average while requiring up to 84% less time to train. We couple our
empirical study with a complexity analysis of the computational cost of
extracting the proposed fractal features, and we study its limitation.

###### keywords:

Fractal geometry, Fractal dimension, Object detection, Deep convolutional
neural networks, Feature extraction,

##  1 Introduction

The concept of fractal geometry was developed by Mandelbrot
mandelbrot1977fractals  to model complex objects in non-integer dimensions. A
significant number of applications have been studying the fractal geometry of
objects in many areas. For instance, the fractal dimension (FD) has been
successfully applied in soil particle-size distribution to provide a unique
quantitative characterization of the soil data spatial distributions and
variability kravchenko1999multifractal . In medicine, FDs have been applied in
the identification of brain tumors from magnetic resonance images
iftekharuddin2003fractal , detection of micro-calcification in mammograms
shanmugavadivu2012fractal , malignancy of skin lesions dobrescu2010medical
and lung tumors mihara1998usefulness . Fractal analysis has also been applied
in industry and production where FDs are used to perform texture and color
analysis and pattern classification to detect defective products xie2008review
. In image segmentation and object detection, fractal models are used to
describe the natural scenes and detect man-made objects from images based on
the fractal difference between artificial targets and natural background
tian2007detection ; wang2016man . In backes2009plant ; florindo2012fractal ,
the fractal descriptors are applied to the analysis of the texture images.
Numerous face recognition applications relied on the fractal theory. For
instance, the FD has been used in pixel-to-pixel face matching kouzani1999face
and face recognition blackledge2009texture . Recently, Joardar et al.
face2019enhanced  presented an enhanced patch-wise feature extraction
technique based on the FD to develop a pose-invariant face recognition system.
However, most of the references discussed above are more than 5 years old and
do not make a clear comparison to DL techniques.

Motivated by the potential that fractal geometry possesses, we investigate the
ability of Convolutional Neural Networks (CNNs) to encode it. This is
currently achieved by Explainable AI (ExAI) researchers through the dissection
of the inner workings of DL models bau2017network ; fong2018net2vec ;
zeiler2014visualizing ; mahendran2015understanding ; dosovitskiy2016inverting
. Measuring similarities between deep representations aids in the
investigation of what DL models are encoding. Canonical Correlation Analysis
(CCA) hardoon2004canonical  has been used to measure similarity between
modeled and measured brain activity sussillo2015neural , to train word
embeddings in multi-lingual language models faruqui2014improving  and to
compare deep representations in an interpretability framework raghu2017svcca .
The centered Kernel Alignment (CKA) method has been later introduced in
kornblith2019similarity  to identify similarities in general conditions.

In this work, we utilize CCA and CKA analysis to show that DL models are not
able to encode the fractal geometry. For this purpose, we propose a method to
extract the fractal dimension at different granularity levels and we conduct a
correlation analysis to prove that DL models are still not able to encode
fractals neither in their early layers where low-level features are usually
extracted nor in deeper levels where higher-level abstractions are formed.

Furthermore, we study the importance of the extracted features in
classification tasks where self-substructure is essential. We show that
training shallow models on the fractal features achieves a performance
comparable to DL, and sometimes even superior in use cases such as
agriculture, remote sensing, and industry while requiring less training time
and computational resources. We augment our study with a complexity analysis,
empirical robustness guarantees, and a human evaluation of the agreement
between DL models and models that operate on the fractal feature solely.

Next, we start by providing a literature review on the network representation
dissection methods and the fractal geometry in Section 2. We then describe how
the fractal features can be extracted from a digital image to be correlated
with the hidden representations of DL models in Section 3. Then, we present
the experimental setup and results in Section 4 before concluding with final
remarks in Section 5.

##  2 Related Work

###  2.1 Feature Encoding in Deep Learning Models

In an attempt to enhance the interpretability of DL models, some researchers
investigated the way patterns are encoded in the hidden layers of a deep
network. Such methods provide a quantitative interpretation of the latent
representations of deep models and study the relevance of specific
features/representations in classification dovsilovic2018explainable .

Network dissection bau2017network , for instance, is a framework that provides
a quantitative interpretation of the latent representations of CNNs by
studying the alignment between CNNs’ hidden units and a set of user-defined
semantic concepts. Layer-wise Relevance Propagation (LRP) bach2015pixel , is a
well-used method in the exAI community where the back-propagated signal is
interpreted as relevant to understanding classification decisions. LRP allows
the visualization of the contributions of the single-pixel in kernel-based
classifiers as heatmaps that can be interpreted and validated by human
experts. Other visualization methods include but are not limited to
zeiler2014visualizing ; mahendran2015understanding ; dosovitskiy2016inverting
.

On the other hand, leino2018influence  propose an influence measure that is
used to explain the essence of a particular class in a classification task and
to identify generally-influential neurons over a distribution. CCA
hardoon2004canonical  has been also used to measure similarity between modeled
and measured brain activity sussillo2015neural , to train word embeddings in
multi-lingual language models faruqui2014improving  and to compare deep
representations in an interpretability framework raghu2017svcca .
kornblith2019similarity  argues that CCA cannot measure meaningful
similarities when the layer’s representation is of a higher dimension than
$n$. kornblith2019similarity  introduce the CKA method closely related to CCA
but can further identify similarities between representations in networks
trained from different initialization.

A rigorous evaluation of these approaches, besides other explainability
methods, has enabled unprecedented breakthroughs in a variety of explainable
computer vision and natural language processing applications tjoa2020survey ;
li2020survey . For instance, CNNs are shown to localize objects without being
explicitly trained on the localization tasks in bazzani2016self . RNNs in
general, and LSTMs and transformers specifically, are also shown to encode
specific syntactic/semantic features in specific layers tenney2019bert .

###  2.2 Fractal Geometry

The concept of fractal geometry was developed by Mandelbrot
mandelbrot1977fractals  to model complex objects in non-integer dimensions. A
significant number of applications have been studying the fractal geometry of
objects in many areas. For instance, the fractal dimension (FD) has been
applied in the detection of micro-calcification in mammograms
shanmugavadivu2012fractal  and malignancy of skin lesions dobrescu2010medical
. Fractal analysis has been also applied in industry and production to perform
texture and pattern classification to detect defective products xie2008review
.

Moreover, the fractal theory has been widely applied in image segmentation and
object detection. In wang2016man , fractal models are used to describe natural
scenes and detect man-made objects from images based on the fractal difference
between artificial targets and natural backgrounds. In florindo2012fractal ,
the fractal descriptors are applied to analyze the images’ texture. Recently,
Joardar et al. face2019enhanced  presented an enhanced patch-wise feature
extraction technique based on the FD to develop a pose-invariant face
recognition system.

It is worth mentioning that given the breakthroughs in DL in the past few
years, the interest in feature extraction in general and fractal features
specifically has significantly declined. This explains why most of the
discussed work here on fractal geometry is more than five years old. Moreover,
the word “fractal” has been recently used in the literature to describe a
design strategy for the macro-architecture of neural networks
larsson2016fractalnet ; zhou2014object ; ning2017knowledge . These networks do
not compute the fractal dimension that was proposed in mandelbrot1977fractals
which is adopted in this work. They, instead, use the term “fractal” to refer
to the residual architecture of their deep networks.

##  3 Methodology

In this work, we are interested in researching the following question: are DL
models able to encode the fractal geometry without explicit training? Thus, we
consider the learned representations in deep models at different layers and we
correlate them to the extracted fractal features using CCA and CKA analyses.
We start by detailing how the fractal feature vector is extracted from a
digital image. Then, we describe the correlation methods that we follow to
study the representation similarity.

###  3.1 Fractal Features

####  3.1.1 Fractal Dimension Computation

The fractal dimension $FD=\frac{\ln(N)}{\ln(1/r)}\in\mathbb{R}$ of an object
is defined as the exponent of the number of self-similar pieces, $N$, with a
magnification factor, $1/r$, into which a figure may be shattered. In a
digital image, $FD$ is estimated using the box-counting method
sarkar1992efficient which estimates the number of boxes with side length $r$
that are needed to cover the surface of a fractal object and the number of
grid boxes $N$ occupied by one or more pixels of the digital image. Once $N$
and $r$ are found, FD is estimated as the slope of the line that best fits the
2D points of the form $(\ln(N),\ln(1/r))$.

####  3.1.2 From Fractal Dimension to Fractal Features

As defined, the fractal dimension is a scalar that reflects the self-structure
in an image. We define the fractal feature vector to be an integration of the
fractal dimension of different sub-images at different granularity levels. For
this purpose, we consider non-overlapping sub-regions of the image of
different sizes $w\times w$, and we compute their fractal dimensions. To
provide robustness to scaling and shifting, we consider different window sizes
$w$, merging all results into one feature vector.

Figure 1 illustrates how the fractal features are computed for a $M\times M$
image by changing the window size from $w=2$ to $w=\frac{M}{2}$, computing the
fractal feature under each $w\times w$ block then flattening and merging all
the results. This formulation enables the study of an object at different
granularity levels by tracking how its internal substructure evolves at
different zoom in levels. Once the vector of the zoomed-in fractal features
(ZFrac) is computed, it is fed into a shallow feed-forward Neural Network
(ZFrac+NN) to perform the classification instead of feeding the raw image.

Figure 1: Computing the fractal features of an $M\times M$ image

###  3.2 Representation Similarity

Formally, a layer representation is the mapping between all possible inputs
and outputs. Since infinite mappings are not practically feasible, we consider
the layer’s representation to be its response over a finite set of inputs
drawn from a particular dataset. Specifically, for a given dataset
$X=\\{\bm{x}_{1},\dots,\bm{x}_{n}\\}$ and a layer $l$, the layer’s
representation is defined by $\bm{Y}\in\mathbb{R}^{n\times
p_{2}}=\big{[}\bm{z_{l}}(\bm{x_{1}})$T$,\dots,\bm{z_{l}}(\bm{x_{n}})$T$\big{]}$Twith
$\bm{z}_{l}(\bm{x}_{i})\in\mathbb{R}^{p_{2}}$ is the concatenation of all the
neurons’ output in $l$ on the input $x_{i}$. Let $Z\in\mathbb{R}^{n\times
p_{1}}$ denote a matrix of fractal features ZFrac for the same $n$ examples.

####  3.2.1 CCA Analysis

Once the fractal features $\bm{Z}$ and the layers representations $\bm{Y}$ are
extracted, CCA method linearly transforms $\bm{Z}$ and $\bm{Y}$ to new sub-
spaces $\bm{\tilde{Z}}=\bm{W_{1}}\bm{Z}$ and
$\bm{\tilde{Y}}=\bm{W_{2}}\bm{Y}l$ to achieve maximum alignment and then
computes their correlation score.

####  3.2.2 CKA Analysis

CKA is a normalized index derived from the Hilbert-Schmidt Independence
Criterion, HSIC, which is computed as follows:

$CKS(Z,Y)=\frac{HSIC(\bm{Z}\bm{Y})}{\sqrt{HSIC(\bm{Y}\bm{Y}).HSIC(\bm{Z}\bm{Z})}}$
(1)

We consider linear and RBF kernels. Based on the recommendation of the authors
in kornblith2019similarity , we set $\sigma$ in the RBF kernel to be a
fraction $\alpha$ of the median distance between the $n$ examples.

CCA and CKA scores are computed across all layers in a DL model. We seek peaks
in the correlation score to localize the layer that is responsible for the
extraction of the feature in question. The absence of peaks suggests the
inability of the network to encode the corresponding feature.

##  4 Experiments

We first show the inability of DL models to encode the fractal feature through
the CCA and the CKA experiments. We complement our study with a human
evaluation to study whether fractal-based and DL models differ for a human
evaluator. Finally, we show the potential of fractal-based networks in
specific classification tasks.

###  4.1 Experimental Setup

For this experiment, an Intel(R) Core(TM) i7 machine with a 4-core CPU at
2.60GHz and 2601 Mhz is considered with Python 3.5  and Keras backend.
ZFrac+NN consists of 2 fully connected layers with 100 and 50 neurons
respectively and a relu activation. Training ZFrac+NN and DL models is done
using Adam optimizer for 200 epochs with an early stopping activated after
each 3-epoch plateau for the validation loss.

We consider 8 different datasets with characteristics summarized in Table 1.
Self-structure is essential for the success of the classification task in the
datasets considered in this work. For instance, the purpose of the bright spot
detection dataset el2019deep is to identify seismic attribute anomalies that
can indicate the presence of hydrocarbon formation. The potato and tomato
disease datasets hughes2015open require recognizing unusual structures in
leaves that implied a plant disease. Similarly, the detection of natural and
industrial defects is needed in the DAGM, steel defect detection, bridge crack
detection xu2019automatic datasets, magnetic tile defect huang2020surface ,
anomaly detection bergmann2019mvtec Electroluminescence defective solar
cellsbuerhop2018benchmark and Kolektor bovzivc2021mixed datasets. Figure 2
shows two sample images from each dataset considered in this work.

Dataset | Classes | # train | # test | Image size
---|---|---|---|---
Bright Spot | 2 | 70 | 30 | 4,017x1,690
Steel Defect111kaggle.com/c/severstal-steel-defect-detection | 4 | 3,500 | 1,500 | 1,600x256
DAGM 222resources.mpi-inf.mpg.de/conference/dagm/2007/prizes.html | 6 | 1,440 | 360 | 512x512
Tomato disease | 2 | 2,780 | 720 | 256x256
Potato disease | 2 | 806 | 346 | 256x256
Bridge Crack | 2 | 4,856 | 1213 | 224x224
Magnetic Defect | 6 | 1,075 | 269 | 224x224
Anomaly Detection | 2 | 3,629 | 1,725 | 1,024x1,024
Kolektor SDD2 | 2 | 2,331 | 1,004 | 230x630
Electroluminescence |  |  |  |
Defective Solar Cells | 2 | 2,099 | 525 | 300x300
Table 1: Experimental datasets description Figure 2: Samples from the datasets
considered in this work. For binary classification datasets, we illustrate one
random image from each category. For multi-class classification datasets, we
choose two categories at random and we visualize one random image from each.
(The images here are resized for display purposes)

###  4.2 Representation Similarity

####  4.2.1 Correlation Results

We consider $150$ randomly chosen samples from the steel defect detection
testing dataset. These samples are then fed to four state-of-the-art (SOTA)
deep models, VGG19 vgg , InceptionV3 inception , ResNet152V2 he2016deep , and
DenseNet201 densenet . In each network, all layer’s activations are recorded
on the $150$ samples that are correlated with the extracted fractal features
at different granularity levels ($2,4,8,16$ and $32$ specifically). The
maximum CCA and CKA scores across all layers are reported in Table 2. We
reinforce our study by computing three other correlation metrics (Pearson,
Spearman, and Kendall) and by reporting the maximum in Table 2. Furthermore,
we compare the correlation between deep representations and the fractal
features to that between deep representations and other low-level features
that are widely adopted in the computer vision community, namely: the Prewitt
edges (horizontal and vertical), Harris corners, Histogram of Oriented
Gradients (HoG), and GLCM texture features.

Metric |  CKA |  CCA |  Pearson |  Spearman |  Kendall |  CKA |  CCA |  Pearson |  Spearman |  Kendall
---|---|---|---|---|---|---|---|---|---|---
Feature | VGG19 | Inception V3
Fractal | 0.1 | 0.08 | 0.02 | 0.05 | 0.04 | 0.09 | 0.06 | 0.04 | 0.04 | 0.01
GLCM | 1.00 | 1.00 | 0.87 | 0.98 | 0.92 | 1.00 | 1.00 | 0.93 | 0.76 | 0.82
Harris | 0.85 | 0.71 | 0.62 | 0.54 | 0.74 | 0.62 | 0.65 | 0.66 | 0.65 | 0.53
HoG | 1.00 | 1.00 | 0.93 | 0.82 | 0.82 | 1.00 | 1.00 | 0.97 | 1.00 | 0.98
Prewitt_H | 0.89 | 0.51 | 0.57 | 0.61 | 0.68 | 0.85 | 0.45 | 0.42 | 0.46 | 0.62
Prewitt_V | 0.87 | 0.99 | 0.97 | 0.89 | 0.97 | 0.81 | 0.91 | 0.97 | 0.99 | 1.00
Feature | ResNet152V1 | DenseNet201
Fractal | 0.06 | 0.04 | 0.1 | 0.09 | 0.1 | 0.09 | 0.06 | 0.03 | 0.07 | 0.09
GLCM | 0.99 | 0.67 | 0.63 | 0.32 | 0.59 | 0.98 | 0.84 | 0.82 | 1.00 | 0.97
Harris | 0.67 | 0.62 | 0.58 | 0.59 | 0.61 | 0.72 | 0.63 | 0.65 | 0.73 | 0.71
HoG | 0.87 | 0.51 | 0.63 | 0.53 | 0.39 | 1.00 | 0.99 | 0.93 | 1.00 | 0.93
Prewitt_H | 0.89 | 0.73 | 0.72 | 0.71 | 0.72 | 0.91 | 0.53 | 0.57 | 0.45 | 0.32
Prewitt_V | 0.93 | 0.59 | 0.64 | 0.82 | 0.53 | 0.96 | 0.82 | 0.78 | 0.93 | 0.8
Table 2: Maximum correlation between the corresponding feature and the hidden
representations of the four DL models. Highest correlations are highlighted in
bold.

As reported in Table 2, the CKA and CCA correlation between the extracted
fractal features and the layer representations was found to be consistently
low ($\leq 0.1$) across all DL models, layers, and correlation metrics. This
low correlation suggests that the DL models that we considered are lacking the
ability to encode the fractal features. Experiments on bright spot detection,
DAGM, and Tomato and Potato disease, show similar results and are reported in
the appendix. The results also reveal that the maximum correlation achieved by
the fractal features is consistently lower than the other low-level features.
The latter can reach a high correlation score (up to 100%) which suggests an
obvious encoding of the corresponding feature in a particular layer. Finally,
as expected, CKA coefficients are consistently higher than those of CCA. This
shows that the CKA method is able to measure more meaningful similarities
between representations as suggested in kornblith2019similarity .

####  4.2.2 Correlation Across Layers

In order to visualize how correlation scores change when the depth of the
network is varied, we plot the CCA scores across different layers for the
fractal feature and the low-level features in the four DL models. The results
in Figure 3 demonstrate that the fractal feature maintains a consistently
negligible correlation with the deep representations of the deep networks
across different layers in comparison to other low-level features according to
their CCA and CKA scores. The graphs also suggest that none of the layers of
the considered deep networks is attempting at extracting the fractal features.
This can be explained by the pre-eminent pattern of the other low-level
features that exhibit a correlation evident in the clear peaks that are
missing in the fractal correlation graph. As a baseline, we visualize the
maximum CCA and CKA between every pair of features in Figure 4. The
correlations show a relatively low association between fractals and the other
low-level features (as compared to the edges (Prewitt) and oriented gradients
(HoG)).

CKA score

fractalsGLCMHarrisHoGPrewitt_hPrewitt_v

VGG19 LayersCCA score

InceptionV3 Layers

ResNet152V2 Layers

DenseNet102 Layers

Figure 3: CKA and CCA scores on different layers in the DL models show
consistently low correlation with the fractal features in comparison to other
low-level features suggesting that such networks are unable to encode the
fractal features.

(a) CKA

(b) CCA

Figure 4: Maximum CKA and CCA correlation score between every pair of features
averaged across all layers on DenseNet201 and the steel defect detection data

####  4.2.3 Agreement

To further investigate the agreement between the fractal dimension and the DL
hidden representations we consider ZFrac+NN and we compare its predictions to
the DL models on the steel defect detection dataset by studying the agreement
percentage between the models. Figure 5 shows that the agreement between
ZFrac+NN and the DL models does not exceed 50% of the test cases while the
four DL models agree on almost all the test cases. One can conclude that, even
with different architectures, the predictions in DL models are almost
consistent; the right, as well as the wrong predictions, agree between
different models. Networks based on the fractal feature however can perceive
images differently; ZFrac+NN can correctly detect some of the defects that DL
models missed and vice versa. This result paves the way to significant
improvement on both approaches when ensembling methods are exploited.

Figure 5: Percentage of agreement on new testing images between ZFrac+NN and
the DL models

###  4.3 Difference between Z-Frac and DL for a human evaluator

We finally conduct a survey on the wrong predictions in ZFrac+NN and DL
models. 140 responses were collected from an audience of different age
groups.333$48\%$ of the participants were female and 60% of the participants
had a prior experience in image classification applications The participants
were asked to evaluate the difficulty of some misclassified images that were
chosen randomly from the experiments we ran on the bright spot detection,
steel defect detection, potato and tomato diseases, and DAGM datasets. Figure
6 shows a sample of the wrong predictions that were included in the survey.
72%, 88%, 92% and 96% and of the participants declared that wrong predictions
in the ZFrac+NN are less obvious than those of the DL models on the DAGM,
potato, steel defect detection, and tomato disease datasets respectively. When
shown 6 misclassified images and asked to choose the hardest 3 images to
classify in each dataset as in Figure 7, participants consistently selected
the top-1 image to belong to the ZFrac+NN model in all datasets, and on
average 2 out of the top-3 images belong to the ZFrac+NN model. The top-3
images selected by the participants to have a disease more obvious to detect
belonged all to the DL wrong predictions in the tomato and potato disease
dataset as shown in Figure 7.

These results emphasize the role the fractal features can play in reflecting
the nature of objects given that ZFrac+NN misses are less obvious than those
missed by DL models. Finally, for the bright spot detection dataset where
ZFrac+NN does not fail on the test set, we asked the participants to rank the
difficulty of the images where DL models failed. 76% and 31% considered the
images as easy and extremely easy respectively which highlights the fact that
DL models can fail badly when limited training data is available even if the
classification task is relatively not challenging.

(a) ZFrac+NN

(b) DL

Figure 6: Sample wrong predictions from the potato disease dataset where 96.5%
of the participants reported that the disease in (b) is more obvious to
detect.

Figure 7: (Top) Sample wrong predictions by ZFrac+NN (a, b and c) and DL
models (d, e and f) where participants are asked to choose the top-3 images
where the leaf disease is obvious to detect. (Bottom) Number of times each
image is selected as having an obvious leaf disease. Top-3 images belong to
wrong predictions in DL models (d, e, and f).

###  4.4 Potential of ZFrac

####  4.4.1 Performance

One could argue that fractal features are not encoded in the presented SOTA
deep networks because they are irrelevant to the classification task. Although
the literature proved the importance of this feature in different
classification tasks, we further compare ZFrac+NN trained on the fractal
features, solely, to the DL models that were pre-trained on ImageNet and fine-
tuned on the raw images. We call the reader’s attention that the goal is not
to show that training on the fractal features solely surpasses training a deep
network on the raw image. We rather aim at highlighting the potential these
features possess for yielding performance comparable to the deep models that
are pre-trained on millions of images.

Table 3 presents the results of training ZFrac+NN and DL models on the raw
images for different datasets. For the bright spot detection dataset, where
limited data is available, ZFrac+NN perfectly predicts bright spots while the
DL performance is very poor, probably because the model is only trained on 70
quite high-dimensional images. On the steel defect detection, potato and
tomato disease, and DAGM datasets where the training data is larger, ZFrac+NN
achieves good results. More specifically, it achieves a performance close to
that of DL in steel defect detection and improves accuracy by up to 47% in the
tomato and potato disease dataset. For the DAGM data, where a multi-class
classification is considered, ZFrac+NN does not achieve the same accuracy as
DL models but consistently takes less time to train by up to a 5x speedup
factor.

Dataset | Model | Acc. | F1 | Training | Avg. Inf.
---|---|---|---|---|---
Bright Spot | ZFrac+NN | 100 | 100 | 106 | 0.143
Detection | VGG19 | 57 | 53 | 4,102 | 0.316
| InceptionV3 | 61 | 64 | 5,344 | 0.273
| ResNet152V2 | 62 | 65 | 5,904 | 0.304
| DenseNet102 | 63 | 61 | 5,827 | 0.315
Steel Defect | ZFrac+NN | 74 | 71 | 7,507 | 0.203
| VGG19 | 71 | 69 | 8,639 | 0.368
| InceptionV3 | 74 | 70 | 1,156 | 0.231
| ResNet152V2 | 67 | 64 | 3,570 | 0.371
| DenseNet102 | 54 | 55 | 2,016 | 0.324
Potato Disease | ZFrac+NN | 88 | 91 | 70 | 0.014
| VGG19 | 54 | 53 | 652 | 0.392
| InceptionV3 | 55 | 47 | 186 | 0.272
| ResNet152V2 | 53 | 42 | 525 | 0.369
| DenseNet102 | 50 | 48 | 378 | 0.307
Tomato Disease | ZFrac+NN | 99 | 98 | 851 | 0.004
| VGG19 | 48 | 53 | 1,750 | 0.312
| InceptionV3 | 52 | 58 | 682 | 0.224
| ResNet152V2 | 50 | 49 | 530 | 0.321
| DenseNet102 | 46 | 47 | 1,008 | 0.315
DAGM | ZFrac+NN | 85 | 82 | 1,699 | 0.007
| VGG19 | 97 | 97 | 9,607 | 0.332
| InceptionV3 | 91 | 92 | 1,864 | 0.295
| ResNet152V2 | 91 | 93 | 6,430 | 0.351
| DenseNet102 | 91 | 92 | 3,841 | 0.324
Bridge Crack | ZFrac+NN | 99 | 97 | 689 | 0.003
| VGG19 | 97 | 96 | 6,902 | 0.360
| InceptionV3 | 97 | 95 | 8,431 | 0.270
| ResNet152V2 | 98 | 93 | 4,544 | 0.315
| DenseNet102 | 97 | 96 | 3,822 | 0.341
Magnetic Defect | ZFrac+NN | 99 | 98 | 85 | 0.004
| VGG19 | 94 | 94 | 1,644 | 0.316
| InceptionV3 | 95 | 94 | 4,852 | 0.273
| ResNet152V2 | 93 | 92 | 5,320 | 0.349
| DenseNet102 | 95 | 93 | 1,024 | 0.325
Anomaly Detection | ZFrac+NN | 94 | 92 | 206 | 0.249
| VGG19 | 89 | 85 | 2,434 | 0.291
| InceptionV3 | 91 | 87 | 4,818 | 0.271
| ResNet152V2 | 89 | 85 | 3,572 | 0.305
| DenseNet102 | 89 | 86 | 3,749 | 0.321
Kolektor SDD2 | ZFrac+NN | 95 | 96 | 891 | 0.004
| VGG19 | 95 | 95 | 1,305 | 0.301
| InceptionV3 | 96 | 93 | 3,481 | 0.281
| ResNet152V2 | 96 | 95 | 3,207 | 0.304
| DenseNet102 | 95 | 94 | 1,984 | 0.325
Electroluminescence | ZFrac+NN | 85 | 87 | 864 | 0.003
Defective | VGG19 | 81 | 78 | 1,634 | 0.298
Solar Cells | InceptionV3 | 85 | 79 | 3,045 | 0.282
| ResNet152V2 | 85 | 79 | 3,011 | 0.306
| DenseNet102 | 85 | 80 | 3,297 | 0.361
Table 3: Performance of ZFrac+NN in comparison to DL models and SOTA models in
terms of accuracy, F1 score (%), training time, and average inference time
(sec). The best results are highlighted in bold.

####  4.4.2 Robustness

When considering anomaly or defect detection tasks, it is unclear how machine
learning models would generalize to data where anomalies manifest themselves
in less significant differences from the training data manifold.

In what follows, we investigate if a fractal feature provides better
generalization guarantees on new data since the nature of the fractal feature
reflects the object’s self-structure. For this purpose, we consider a new
dataset related to disaster detection as in mouzannardamage . In a digital
image, a disaster can be defined as a massive perturbance in the structure of
objects which could be well studied in the fractal geometry framework. To test
the robustness of our approach, we train ZFrac+NN and 4 DL models that are
initialized by random weights - for a fair comparison, on images labeled as
damage and non-damage. The damage class has diverse categories ranging from
earthquakes to floods and hurricanes. During testing, we consider images
showing damage as fire, which was never encountered during training. Table 4
shows that ZFrac+NN not only achieves better performance on small datasets but
it generalizes well on data distributions not encountered during training.
This can be explained by the fact that the fractal features are reflecting the
sub-structures in the image, which can directly relate to chaos and disaster
detection. ZFrac+NN learns how to detect defects faster and more robustly.

Number of training images | 200 | 400 | 800
---|---|---|---
ZFrac+NN | 65 | 71 | 75
VGG19 | 42 | 43 | 47
InceptionV3 | 44 | 46 | 59
ResNet152V2 | 47 | 49 | 61
DenseNet102 | 51 | 53 | 61
Table 4: Accuracy (%) when testing on fire category

####  4.4.3 Complexity

Studying the computational cost of the extraction of fractal features has a
great impact on the efficiency of our method. One could argue that the
computation grows exponentially with the number of space coordinates. However,
when dealing with $M\times M$ images, computing FD of $(\frac{M}{w})^{2}$ sub-
images of size $w\times w$ takes a number of FLOPS of order
$(\frac{M}{w})^{2}$. Thus, for $\log M$ windows, the number of FLOPs is of
order $M^{5}\log M$, which is computationally cheap given that $M$ is less
than $1000$ in most of the applications. One can also note that FD computation
in a sub-image is independent of others, thus it is easily parallelizable,
which can further speed up the training and inference process. Finally,
processing the fractal features by a shallow network requires $<10$ MB of
memory storage whereas DL models require $>300$MB which might not be always
feasible on older generation hardware or for some real-life applications with
limited resources such as IoT. Figure 8 shows the training and inference times
of Table LABEL:tbl:main aggregated across datasets shows that training
ZFrac+NN is more consistent than that of DL models where there is a low
variance in the training time. When it comes to the deployed models, the
inference time for ZFrac+NN is consistently less than that of the DL models
however with higher variance. This can be explained by the fact that ZFrac+NN
does not require an image resize. Given that the sizes of the images in the
datasets considered in this work vary from 224x224 to 1,024x1,024, the time
needed to extract the fractal feature vector will vary considerably. DL models
which operate on 224x224 or 229x229 images mostly require a consistent
inference time. For applications where a fast inference is crucial, images can
be always resized for deployed models.

Figure 8: Training and inference time of ZFrac+NN and the four DL models

###  4.5 Limitations

FD aims at describing the self-structure of objects which makes it more
suitable for binary classification of defect detection in particular. General
classification such as the work on CIFAR and ImageNet data requires more than
self-substructure such as colors, context knowledge, and object abstraction.
For instance, ZFrac+NN achieves a 67, 35 and 15% accuracy on MNIST, PETA, and
CIFAR, datasets tang2019improving ; byerly2020branching ; kolesnikov2019big
while the SOTA performance is 99, 80, and 99% respectively. This can be
explained by the nature of the datasets and the great efforts that have been
put into optimizing the performance of DL models. For instance, the PETA data
is of low resolution and was taken at a far distance. In such cases, the
magnification process in the fractals computation fails to identify
substructures, which leads to a weak image understanding, thus poor results.
Moreover, the classification of CIFAR and MNIST data requires complex and
high-level features that are not included in ZFrac+NN.

One can thus conclude that the fractal feature has great potential in
classification tasks where the self-structure is the essence of that task such
as the case of defect detection. However, considered solely, the fractal
feature has limitations in classification tasks where higher cognitive
knowledge is required.

##  5 Conclusion

In this work, we inspected deep networks’ performance within the fractal
geometry framework and we showed that DL models are unable to extract fractal
features across different depth levels. We empirically demonstrate that
shallow networks trained on the fractal feature exhibit performance comparable
to that of DL models that are trained on raw images while requiring less
training time in specific tasks. In addition to that, given that fractal
geometry studies the internal structure of complex objects, we reveal that
such features can lead to better and more robust results in tasks where the
study of the object substructure is crucial for classification. We also
provide a theoretical complexity analysis showing that computing the fractal
feature is efficient. We finally augment our study with a human evaluation of
the wrong predictions by DL models as well as ZFrac+NN. The human evaluation
showed that, on average, classifications missed by the fractal features are
perceived as “not obvious” and harder to detect than those of DL models. A
subsequent step in this line of work can focus on utilizing the fractal
geometry in DL through architecture adaptation.

## References

  *   * (1) B. Mandelbrot, _Fractals_ (Freeman San Francisco, 1977)
  * (2) A.N. Kravchenko, C.W. Boast, D.G. Bullock, Multifractal analysis of soil spatial variability.  Agronomy Journal 91(6), 1033–1041 (1999)
  * (3) K.M. Iftekharuddin, W. Jia, R. Marsh, Fractal analysis of tumor in brain mr images.  Machine Vision and Applications 13(5-6), 352–362 (2003)
  * (4) P. Shanmugavadivu, V. Sivakumar, Fractal dimension based texture analysis of digital images.  Procedia engineering 38, 2981–2986 (2012)
  * (5) R. Dobrescu, M. Dobrescu, S. Mocanu, D. Popescu, Medical images classification for skin cancer diagnosis based on combined texture and fractal analysis.  WISEAS Transactions on Biology and Biomedicine 7(3), 223–232 (2010)
  * (6) N. Mihara, K. Kuriyama, S. Kido, C. Kuroda, T. Johkoh, H. Naito, H. Nakamura, The usefulness of fractal geometry for the diagnosis of small peripheral lung tumors.  Nihon Igaku Hoshasen Gakkai zasshi. Nippon acta radiologica 58(4), 148–151 (1998)
  * (7) X. Xie, A review of recent advances in surface defect detection using texture analysis techniques.  ELCVIA 7(3), 1–22 (2008)
  * (8) B. Tian, J. Yuan, J. Tian, H. Liang, in _MIPPR 2007: Automatic Target Recognition and Image Analysis; and Multispectral Image Acquisition_ , vol. 6786 (International Society for Optics and Photonics, 2007), p. 67861R
  * (9) Z. Wang, X. Shen, W. Xia, Q. Yu, in _Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), International Congress on_ (IEEE, 2016), pp. 489–493
  * (10) A.R. Backes, D. Casanova, O.M. Bruno, Plant leaf identification based on volumetric fractal dimension.  International Journal of Pattern Recognition and Artificial Intelligence 23(06), 1145–1160 (2009)
  * (11) J.B. Florindo, O.M. Bruno, Fractal descriptors based on fourier spectrum applied to texture analysis.  Physica A: statistical Mechanics and its Applications 391(20), 4909–4922 (2012)
  * (12) A.Z. Kouzani, F. He, K. Sammut, in _Image Processing, 1999. ICIP 99. Proceedings. 1999 International Conference on_ , vol. 3 (IEEE, 1999), pp. 642–646
  * (13) J. Blackledge, D.A. Dubovitskiy, in _Theory and Practice of Computer Graphics_ (The Eurographics Association, 2009)
  * (14) S. Joardar, A. Sanyal, D. Sen, D. Sen, A. Chatterjee, in _Decision Science in Action_ (Springer, 2019), pp. 217–226
  * (15) D. Bau, B. Zhou, A. Khosla, A. Oliva, A. Torralba, in _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2017), pp. 6541–6549
  * (16) R. Fong, A. Vedaldi, in _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2018), pp. 8730–8738
  * (17) M.D. Zeiler, R. Fergus, in _European conference on computer vision_ (Springer, 2014), pp. 818–833
  * (18) A. Mahendran, A. Vedaldi, in _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2015), pp. 5188–5196
  * (19) A. Dosovitskiy, T. Brox, in _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2016), pp. 4829–4837
  * (20) D.R. Hardoon, S. Szedmak, J. Shawe-Taylor, Canonical correlation analysis: An overview with application to learning methods.  Neural computation 16(12), 2639–2664 (2004)
  * (21) D. Sussillo, M.M. Churchland, M.T. Kaufman, K.V. Shenoy, A neural network that finds a naturalistic solution for the production of muscle activity.  Nature neuroscience 18(7), 1025–1033 (2015)
  * (22) M. Faruqui, C. Dyer, in _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ (2014), pp. 462–471
  * (23) M. Raghu, J. Gilmer, J. Yosinski, J. Sohl-Dickstein, in _Advances in Neural Information Processing Systems_ (2017), pp. 6076–6085
  * (24) S. Kornblith, M. Norouzi, H. Lee, G. Hinton, in _International Conference on Machine Learning_ (PMLR, 2019), pp. 3519–3529
  * (25) F.K. Došilović, M. Brčić, N. Hlupić, in _2018 41st International convention on information and communication technology, electronics and microelectronics (MIPRO)_ (IEEE, 2018), pp. 0210–0215
  * (26) S. Bach, A. Binder, G. Montavon, F. Klauschen, K.R. Müller, W. Samek, On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PloS one 10(7), e0130,140 (2015)
  * (27) K. Leino, S. Sen, A. Datta, M. Fredrikson, L. Li, in _2018 IEEE International Test Conference (ITC)_ (IEEE, 2018), pp. 1–8
  * (28) E. Tjoa, C. Guan, A survey on explainable artificial intelligence (xai): Toward medical xai.  IEEE transactions on neural networks and learning systems 32(11), 4793–4813 (2020)
  * (29) X.H. Li, C.C. Cao, Y. Shi, W. Bai, H. Gao, L. Qiu, C. Wang, Y. Gao, S. Zhang, X. Xue, et al., A survey of data-driven and knowledge-aware explainable ai.  IEEE Transactions on Knowledge and Data Engineering 34(1), 29–49 (2020)
  * (30) L. Bazzani, A. Bergamo, D. Anguelov, L. Torresani, in _2016 IEEE winter conference on applications of computer vision (WACV)_ (IEEE, 2016), pp. 1–9
  * (31) I. Tenney, D. Das, E. Pavlick, Bert rediscovers the classical nlp pipeline.  arXiv preprint arXiv:1905.05950 (2019)
  * (32) G. Larsson, M. Maire, G. Shakhnarovich, Fractalnet: Ultra-deep neural networks without residuals.  arXiv preprint arXiv:1605.07648 (2016)
  * (33) B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Object detectors emerge in deep scene cnns.  arXiv preprint arXiv:1412.6856 (2014)
  * (34) G. Ning, Z. Zhang, Z. He, Knowledge-guided deep fractal neural networks for human pose estimation.  IEEE Transactions on Multimedia 20(5), 1246–1259 (2017)
  * (35) N. Sarkar, B.B. Chaudhuri, An efficient approach to estimate fractal dimension of textural images.  Pattern recognition 25(9), 1035–1041 (1992)
  * (36) J. El Zini, Y. Rizk, M. Awad, A deep transfer learning framework for seismic data analysis: A case study on bright spot detection.  IEEE Transactions on Geoscience and Remote Sensing (2019)
  * (37) D. Hughes, M. Salathé, et al., An open access repository of images on plant health to enable the development of mobile disease diagnostics.  arXiv preprint arXiv:1511.08060 (2015)
  * (38) H. Xu, X. Su, Y. Wang, H. Cai, K. Cui, X. Chen, Automatic bridge crack detection using a convolutional neural network.  Applied Sciences 9(14), 2867 (2019)
  * (39) Y. Huang, C. Qiu, K. Yuan, Surface defect saliency of magnetic tile.  The Visual Computer 36(1), 85–96 (2020)
  * (40) P. Bergmann, M. Fauser, D. Sattlegger, C. Steger, in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2019), pp. 9592–9600
  * (41) C. Buerhop-Lutz, S. Deitsch, A. Maier, F. Gallwitz, C. Brabec, in _35th European PV Solar Energy Conference and Exhibition_ , vol. 12871289 (2018)
  * (42) J. Božič, D. Tabernik, D. Skočaj, Mixed supervision for surface-defect detection: from weakly to fully supervised learning.  Computers in Industry 129, 103,459 (2021)
  * (43) K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 (2014)
  * (44) C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, in _Proceedings of the IEEE CVPR_ (2016), pp. 2818–2826
  * (45) K. He, X. Zhang, S. Ren, J. Sun, in _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2016), pp. 770–778
  * (46) G. Huang, Z. Liu, L. Van Der Maaten, K.Q. Weinberger, in _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2017), pp. 4700–4708
  * (47) H. Mouzannar, Y. Rizk, M. Awad, in _International Conference on Information Systems for Crisis Response and Management_ (Rochester, New York, USA, 2018)
  * (48) C. Tang, L. Sheng, Z. Zhang, X. Hu, in _Proceedings of the IEEE ICCV_ (2019), pp. 4997–5006
  * (49) A. Byerly, T. Kalganova, I. Dear, A branching and merging convolutional network with homogeneous filter capsules.  arXiv preprint arXiv:2001.09136 (2020)
  * (50) A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, N. Houlsby, Big transfer (bit): General visual representation learning.  arXiv preprint arXiv:1912.11370 (2019)