11institutetext: Indian Institute of Technology, Madras 22institutetext:
University of Illinois Urbana-Champaign 33institutetext: Rizzo International,
Inc., USA

# CrackSeg9k: A Collection and Benchmark for Crack Segmentation Datasets and
Frameworks

Shreyas Kulkarni 11 Shreyas Singh 11 Dhananjay Balakrishnan 11 Siddharth
Sharma 11 Saipraneeth Devunuri 22 Sai Chowdeswara Rao Korlapati 33

###### Abstract

The detection of cracks is a crucial task in monitoring structural health and
ensuring structural safety. The manual process of crack detection is time-
consuming and subjective to the inspectors. Several researchers have tried
tackling this problem using traditional Image Processing or learning-based
techniques. However, their scope of work is limited to detecting cracks on a
single type of surface (walls, pavements, glass, etc.). The metrics used to
evaluate these methods are also varied across the literature, making it
challenging to compare techniques. This paper addresses these problems by
combining previously available datasets and unifying the annotations by
tackling the inherent problems within each dataset, such as noise and
distortions. We also present a pipeline that combines Image Processing and
Deep Learning models. Finally, we benchmark the results of proposed models on
these metrics on our new dataset and compare them with state-of-the-art models
in the literature.

###### Keywords:

Deep Learning Applications, Image Processing, Semantic Segmentation, Crack
Detection, Datasets , DeepLab , CFD

##  1 Introduction

Cracks are common building distress, which is a potential threat to the safety
and integrity of the buildings. Localizing and fixing the cracks is a major
responsibility in maintaining the building. However, the task of detecting
cracks is both tedious and repetitive. To expedite this process and alleviate
the workload on experts, it is necessary to achieve automation in crack
detection and segmentation.

For a long time, such crack detection has been done manually. Of late, much
research has gone into developing automated techniques. Initial work focused
on using image thresholding and edge detection techniques [11] [17] [45] [1].
Recently, focus has shifted to using Deep Learning for classification [41]
[42] [12] [23], object detection [11] [30] [33] [27] and semantic segmentation
[46] [26] [50] [40] of cracks. Attempts have been made to use various
techniques, including feature pyramid networks [48] and segmentation models
like U-Net [24]. Some experiments have been conducted using pre-trained Conv-
Nets [52], with limited crack-detection abilities. Better results were
obtained when crack segmentation was attempted using models from the R-CNN
family [20].

The Deep Learning approaches are highly dependent on the data that has been
used for training. The study by [13] makes use of 3 datasets: AigleRN [4] (38
images), CFD dataset [21] (118 images), and the HTR dataset (134 images, not
publicly available). Another work by [25] performs analysis on the CFD dataset
[21] and Crack500 dataset [47]. [31] have evaluated their pipeline on two
datasets containing 56 grayscale and 166 RGB images respectively. SDDNet [6]
(200 images) manually created a dataset to check their model’s performance.
DeepCrack [28] too highlights the lack of a benchmark dataset and makes an
effort to create a dataset of 537 RGB images, which has been included in our
dataset.

Figure 1: Inconsistent Annotations among sub-datasets

All the attempts mentioned above have fallen short of requirements primarily
due to the quality and quantity of publicly available data sets. Obtaining an
annotated data set with segmented masks for cracks is a challenge. Most crack
data sets have less than 500 images, with several having less than 100 images.
Further, the data set being used is not generalizable, and this, combined with
the quality of the data sets, leads to non-reproducible results. The datasets
used in all cases have different resolutions of the images and different types
of annotations leading to inconsistencies as shown in Fig. 1. This makes it
more difficult to compare with other works. We work towards unifying all the
sub-datasets involved to deal with such issues.

We aim to create an efficient Crack segmentation model using the latest
Computer Vision and Deep Learning advances. While we did perform bounding box
prediction to localize crack regions in images of arbitrary resolutions, our
primary focus has remained on binary segmentation of the corresponding cracks.
To overcome the limiting factor of the quality of data sets, we combined data
sets from various sources while also attempting to unify their ground truths
using Image Processing techniques( detailed in Section 3.2). This paper makes
the following major contributions:

  * •

Compilation of a dataset consisting of 9000+ images of cracks on diverse
surfaces with annotations for semantic segmentation. The compiled data is also
categorized into linear, branched, and webbed.

  * •

Further refinement of the ground truth masks by denoising and unifying the
annotations using Image Processing techniques per the dataset’s requirements.

  * •

An end-to-end pipeline has been devised for efficient localization and
segmentation of cracks for images of arbitrary lighting, orientation,
resolution and background.

  * •

We explore various rule-based and data-driven techniques for crack
segmentation to establish a benchmark on our dataset.

  * •

Finally, we present an approach to fuse unsupervised self-attention mechanisms
into CNNs for improvising over the SOTA models.

##  2 Related Works

###  2.1 Rule based Image Processing Techniques

Rule-based Image Processing methods have been traditionally used for image-
based crack segmentation. The techniques can broadly be classified into two
categories, i.e., Edge detection and Thresholding [22].

Thresholding complies with the rule of classifying the pixels of a given image
into groups of objects and background; the former includes those pixels with
gray values more than a specific threshold, and the latter comprises those
with gray values equal to and below the threshold. Choosing an appropriate
threshold value is the main challenge in this method. Using a histogram to
choose a threshold value gives satisfactory results for segmenting images of
road surfaces [29].

The purpose of the edge-based algorithm is to segment an image based on the
discontinuity detection technique [43]. The focus of edge-based algorithms is
on the linear features that mainly correspond to crack boundaries and
interesting object surface details. Based on the comparative study, the
algorithms calculate the magnitude for each pixel and use double Thresholding
to discover potential edges. Even though useful for edge detection, these
algorithms can only detect edges and cannot extrapolate over the entire crack
surface; this makes edge detection ineffective for binary segmentation.

###  2.2 Data Driven Methods

The lack of generalizability of rule-based methods to diverse environments
with different backgrounds and lighting has promoted research into the data-
driven detection of cracks. These techniques comprise of data-hungry Deep
Learning models that are either fully convolutional, attention based or a mix
of both, requiring the training of millions of parameters.

While traditional object detection algorithms re-purposed classifiers to
perform detection, YOLO[35] framed object detection as a regression problem to
spatially separated bounding boxes and associated class probabilities. A
single neural network directly predicts bounding boxes and class probabilities
from full images in one evaluation. We have trained a YOLO-v5 object detector
to localize cracks in images of arbitrary size to crop them before pixel-wise
segmentation.

The image-to-Image translation is the spatial transformation of images from
one domain to another. This field has grown immensely since Pix2Pix GAN [18]
was introduced. GANs [14] consist of two parts: a generative model $G$ that
captures the data distribution and a discriminative model $D$ that estimates
the probability that a sample came from the training data rather than $G$. The
training procedure for $G$ is to maximize the probability of $D$ making a
mistake. We pose semantic segmentation of cracks using Pix2Pix as an image-to-
image translation problem, where the generator translates an input RGB image
into a binary segmentation mask for cracks, and the discriminator tries to
distinguish b/w real and fake samples conditioned on the input. The U-Net[37]
architecture is used for semantic segmentation, which consists of a
contracting path to capture context and a symmetric expanding path that
enables precise localization. This has been utilized as a backbone for the
Generator of GANs.

DeepLab[5] highlights convolution with upsampled filters, or ‘atrous
convolution’ and ASPP ‘Atrous Spatial Pyramidal Pooling’, as a powerful tool
in dense segmentation tasks. DMA-Net [44] proposes a multi-scale attention
module in the decoder of DeepLabv3+ to generate an attention mask and
dynamically assign weights between high-level and low-level feature maps for
crack segmentation.

Swin-Unet[2] is an U-Net-like pure Transformer for image segmentation. The
tokenized image patches are fed into the Transformer-based U-shaped Encoder-
Decoder architecture with skip-connections for local-global semantic feature
learning. Specifically, a hierarchical Swin Transformer with shifted windows
is used as the encoder to extract context features. Moreover, a symmetric Swin
Transformer-based decoder with a patch expanding layer is designed to perform
the up-sampling operation to restore the spatial resolution of the feature
maps.

DINO [3] highlights that self-supervised Vision Transformer features contain
explicit information about the semantic segmentation of an image, which does
not emerge as clearly with supervised Vision Transformers, nor with
Convolutional Nets. Self-supervised ViT features explicitly contain the scene
layout and, in particular, object boundaries. This information is directly
accessible in the self-attention modules of the last block.

##  3 Dataset

The proposed dataset is the largest, most diverse and consistent crack
segmentation dataset constructed so far. It contains a total of 9255 images,
combining different smaller open source datasets. Therefore, the dataset has
great diversity in the surface, background, lighting, exposure, crack width
and type (linear, branched, webbed, non-crack). Smaller datasets like Eugen
Muller [15] and Sylvie Chambon [4] were discarded due to poor quality and much
fewer images. The DIC dataset [36] containing 400+ images was avoided because
the images were augmented from a tiny set of 17 images and had very little
variety. The Forest dataset [39] was excluded as it was found to be very
similar to the CFD dataset.

This dataset is divided into training, validation and testing sets with 90%,
5% and 5% split. The final dataset is made available to public on Harvard
Dataverse[38].

###  3.1 Dataset Details

Table 1 summarises the sub-datasets involved. This subsection will discuss the
sub-datasets involved by providing an overview of data collection techniques,
characteristics of image and dataset size.

Table 1: Sub dataset quantitative details and performance on benchmark models

Dataset | Size | Resolution | % of | Pix2Pix | Deeplab
---|---|---|---|---|---
|  |  | cracks | U-Net | ResNet *
|  |  |  | mIoU | F1 | mIoU | F1
Masonry | 240 | 224 $\times$ 224 | 4.21 | 0.4685 | 0.0392 | 0.4986 | 0.0420
CFD | 118 | 480 $\times$ 320 | 1.61 | 0.6100 | 0.3942 | 0.6232 | 0.4203
CrackTree200 | 175 | 800 $\times$ 600 | 0.31 | 0.5030 | 0.0799 | 0.5478 | 0.1132
Volker | 427 | 512 $\times$ 512 | 4.05 | 0.6743 | 0.5423 | 0.8209 | 0.7955
DeepCrack | 443 | 544 $\times$384 | 3.58 | 0.7207 | 0.6193 | 0.8311 | 0.8068
Ceramic | 100 | 256 $\times$ 256 | 2.05 | 0.4783 | 0.0330 | 0.5095 | 0.0480
SDNET2018 | 1411 | 4032 $\times$ 3024 | 0 | 0.4865 | N/A | 0.5000 | N/A
Rissbilder | 2736 | 512 $\times$ 512 | 2.70 | 0.6050 | 0.3854 | 0.7512 | 0.6856
Crack500 | 3126 | 2000 $\times$ 1500 | 6.03 | 0.6495 | 0.4974 | 0.8230 | 0.8032
GAPS384 | 383 | 640 $\times$ 540 | 1.21 | 0.5716 | 0.2825 | 0.5965 | 0.3358

The Masonry[8] dataset contains images of masonry walls. They consist of
images from the Internet and masonry buildings in the Groningen, Netherlands.
The crack patches comprise of small (a couple of bricks) to larger (whole
masonry walls) fields of view.

CFD[21] dataset is composed of images reflecting urban road surface conditions
in Beijing, China. Each image has hand-labelled ground truth contours.
CrackTree200[51], Volker[32] and DeepCrack[28] contain images with manually
annotated cracks collected from pavements and buildings. They have challenges
like shadows, occlusions, low contrast and noise.

Ceramic Cracks Database[19] was collected by students of the University of
Pernambuco. The database has images of building facades with cracks on
different colours and textures of ceramics.

Figure 2: Refined Dataset. Hairline distortions have been tackled in the lower
half of the images while the thickness has been reduced in the upper half of
the images

SDNET2018[9] contains various non-crack images from concrete bridge decks,
walls, and pavements. The dataset includes cracks as narrow as 0.06 mm and as
wide as 25 mm. The dataset also includes images with various obstructions,
including shadows, surface roughness, scaling, edges, holes, and background
debris.

Rissbilder[32] consists of varied types of architectural cracks, and
Crack500[47] was collected at the Temple University campus using a smartphone
as the data sensor. The GAPS[10] data acquisition occurred in the summer of
2015, so the measuring condition was dry and warm.

###  3.2 Dataset Refinement

Table 2: Sub dataset Refinement Summary Dataset | Distortion | Refinement
---|---|---
Masonry | None | None
CFD | None | None
CrackTree | Continuity errors; | Dilation followed by
200 | Hairline distortion | Erosion with special kernels
Volker | Randomly thick cracks; | Contour area based erosion;
| Hairline distortion | Morph opening
DeepCrack | Crack boundary irregular | None
Ceramic | None | None
SDNET2018 | None | None
Rissbilder | Hairline distortion | Morph opening with
for Florian |  | increased kernel size
Crack500 | Extra thick and | Erosion and
| hairline distortion | morph opening
GAPS384 | Random thickening | Erosion or Dilation on
| of cracks | the basis of Contour size

The open source datasets had different resolutions and the ground truths had
many distortions and discontinuities. They were also vastly inconsistent in
their thickness(see Fig.1). Many cracks had a hairlike boundary noise
surrounding them. This led to scenarios where the model’s metric score was
less even for visually accurate predictions due to the distortions in the
Ground Truths.

Firstly, we resize all the images to a standard size of 400$\times$400\. For
images greater than this size, random crops of this size were extracted from
images, and images with resolution less than it were linearly interpolated to
the standard size. Once the uniformity in size was attained, we performed
Gaussian blurring followed by morphological operations to remove noise from
the image. A 3x3 kernel made up of ones was used to erode the mask, followed
by dilation with the hairline distortions to remove it. Depending on the
distortion (thickened or thinned), the images required either erosion or
dilation. We used the contour area as the metric for finding the kind of
distortion and operating on it, respectively. We have summarised all the
transformations performed on the sub-datasets in the Table 6 and sample
refinements can be visualised in Fig.2.

###  3.3 Categorizing Dataset based on Crack Types

The dataset has been divided into three subtypes, namely Linear, Branched and
Webbed, based on the spatial patterns of cracks observed in the images. To
achieve the classification for the entire dataset, we trained a small
ResNet-18 pre-trained classifier on 150 images, with 50 belonging to each
class. We manually labelled these 150 images to generate ground truths for
classification. This methodology generalized well to our entire dataset.

Figure 3: Sample ground truths belonging to each category of crack type

The cracks with no branches are classified as linear, Cracks with 2-5 branches
are classified as branched, and cracks with more than five cracks are
classified as webbed. This division of the dataset aims to benchmark and
contrast the generalizing ability of benchmarked models across various crack
modalities and complexity for detection. Fig. 3 shows the cracks belonging to
different classes, and Table 3 contains details of the crack types and their
distribution in the dataset.

Table 3: performance based on the type of crack Dataset | Size | Percentage | Pix2Pix | Deeplab
---|---|---|---|---
|  | of cracks | U-Net | ResNet *
|  |  | mIoU | F1 | mIoU | F1
Linear | 2369 | 4.94 | 0.6687 | 0.5455 | 0.8219 | 0.7991
Branched | 4192 | 5.45 | 0.6184 | 0.4374 | 0.7940 | 0.7643
Webbed | 1283 | 9.81 | 0.5963 | 0.4259 | 0.6488 | 0.5835
Non-crack | 1411 | 0 | 0.4865 | N/A | 0.5000 | N/A

##  4 Methodology & Experiments

This section will discuss the experiments conducted to benchmark and evaluate
different models for crack segmentation. These experiments included an
amalgamation of both supervised and unsupervised techniques. The experiments
were performed on the refined dataset with a standard resolution of 400
$\times$ 400 for all images.

###  4.1 Pipeline

Figure 4: General Pipeline

This subsection explains the details of our pipeline for end-to-end crack
segmentation. The pipeline processes the input image through four steps (see
Figure 4). The first step consists of a YOLO-v5-based object detector that
localizes the cracks in an input image of arbitrary resolution and draws
bounding boxes for the cracks in the image. We create bounding box labels for
object detection enclosing the crack using the extreme pixels of the segmented
masks (ground truths). In the second step, cracks are cropped using the
bounding box coordinates obtained in the previous step and resized to
400$\times$400\. The resized crack images are passed through the DINO model to
extract unsupervised features that will be used as the prior for the
segmentation model.

In the final step, the unsupervised features are concatenated with the resized
crack image and fed into an FPN-based binary segmentation model to get pixel-
wise classification for the cracks in each image. The final post-processing
step passes the segmentation maps through a denoiser; the denoised
segmentation mask is reshaped to the size of its original bounding box.
Finally, all the predictions for a given region are stitched together to
produce the final output. The results on various techniques have been
benchmarked in Table 2 and images with their predictions on the respective
models are showcased in Fig.5. Note: the purpose of the YOLO object detector
is to concentrate the segmentation model’s focus on a patch of cracks in case
of high-resolution images.

Table 4:  Model Performance Summary 222* - pre-training on ImageNet. $\checkmark$ \- DINO features used as prior for training Technique | Backbone | Trainable | F1 Score | MIoU
---|---|---|---|---
|  | Parameters |  |
|  | in Millions |  |
Otsu | None | 0 | 0.0609 | 0.3187
DINO | ViT-S | 0 | 0.1568 | 0.5149
MaskRCNN | ResNet-50 | 44.178 | 0.4761 | 0.5213
SWIN Transformer | U-Net | 41.380 | 0.5009 | 0.6426
Pix2Pix | U-Net | 54.421 | 0.5652 | 0.6666
Pix2Pix $\checkmark$ | U-Net | 54.422 | 0.5824 | 0.6953
Pix2Pix | FPN-ResNet-50 | 26.911 | 0.3950 | 0.6286
Pix2Pix $\checkmark$ | FPN-ResNet-50 | 26.914 | 0.5765 | 0.7142
DeepLab V3+ | Xception * | 54.700 | 0.6344 | 0.7208
DeepLab V3+$\checkmark$ | Xception * | 54.700 | 0.6608 | 0.7353
DeepLab V3+ | ResNet-101 * | 59.339 | 0.7060 | 0.7599
DeepLab V3+$\checkmark$ | ResNet-101 * | 59.342 | 0.7238 | 0.7712

###  4.2 Unsupervised Feature Extraction

In our first set of experiments, we attempt to generate the predicted masks
without utilizing ground truth annotations.

Otsu Thresholding method failed to find the optimum threshold for images with
noise. When the dimensions of the cracks were small relative to the
background, the technique failed to identify the cracks. Otsu’s thresholding
method was also unsuccessful in effectively segmenting cracks when the images
were taken under non-uniform lighting conditions.

DINO for ViT We used the ViT-Small model trained in a fully unsupervised
fashion using fully self-supervised DINO [3] which generates self-attention
maps for the input images. The generated maps were found to be very noisy, but
the self-attention maps captured the branched cracks very successfully due to
the global self-attention mechanism of Vision Transformers. Unlike Vision
Transformers, CNN-based algorithms lack the global field of view and are
dataset antagonistic due to weight sharing, due to which CNN-based algorithms
underperform on branched and webbed cracks. The global attention maps
generated by transformers via self-supervision will be further utilized in
Section 4.4 to generate priors for CNN-based segmentation algorithms to
mitigate the problems mentioned above.

Figure 5: Benchmarked Model Performance Comparisons on Refined Dataset

###  4.3 Supervised Techniques

The unsupervised techniques gave noisy results and failed to capture the
domain essence. Therefore, these techniques could not be used directly to
infer results and merely act as priors for supervised techniques. In an
attempt to improve the results, we conducted experiments using supervised
techniques. Supervised techniques involve the utilization of segmented ground
truths and they are more robust than unsupervised methods for this reason. We
used four supervised Deep Learning algorithms for Image segmentation:
MaskRCNN, DeepLab, Pix2Pix and Swin-Unet.

We trained MaskRCNN model using a ResNet-50 [16] backbone. The results
obtained are significantly better than that of unsupervised techniques.
However, we noticed two major issues with Mask-RCNN, the model does not
generalize well to the thickness of the cracks despite the diversity in the
dataset and is very susceptible to background noise such as debris and
shadows, which led to a lot of false positives.

Swin-Unet [2] model was used on our dataset to measure the performance of
transformer-based backbones trained end to end in a supervised fashion. The
Swin Transformer blocks can model the long-range dependencies of different
parts in a crack image from a local and global perspective, which leads to
superior performance in the detection of branched and webbed crack.

We conducted two different experiments with Pix2Pix model, using Vanilla U-Net
with skip connections and Feature Pyramidal Network with a resnet-50 backbone
as the generators. The Patch GAN discriminator used in the Pix2Pix model
improves the quality of predictions significantly. The models are able to
detect finer cracks and the boundaries of cracks with higher precision. The
experiments also highlight that Feature pyramidal networks are superior for
segmentation tasks than U-Nets, as they achieve higher accuracy with less than
half the number of parameters of U-Nets.

DeepLab V3+ using ResNet-101 and Xception [7] backbones pre-trained on
imagenet as encoders. DeepLab is able to detect very fine-grained cracks even
in adverse lighting conditions, the predicted masks generalize well to the
thickness of the cracks, and in some cases, the model was able to detect
cracks that were not annotated in the ground truth. The qualitative and
quantitative results indicate DeepLab as the best model among our experiments.

Figure 6: Transformation in predictions when features are integrated

###  4.4 Using Semi-Supervised DINO features as prior for segmentation

We conducted this experiment to verify our hypothesis that using semi-
supervised DINO features prior to CNN-based image segmentation models leads to
superior results on branched and webbed cracks. This is because global and
local self-attention from transformers helps in establishing long-range
dependencies in images and CNNs help in learning dataset-specific localized
features. For this experiment, we concatenated the semi-supervised self-
attention map generated from DINO as the fourth channel to our input image
tensor. We conducted this experiment with all our CNN-based models and
observed the following: FPN Pix2Pix(14% mIoU increase), U-Net Pix2Pix(4% mIoU
increase), Resnet-101 DeepLab (2% mIoU increase) and Xception DeepLab (1% mIoU
increase). To highlight the improvement in performance due to features, please
refer Fig. 6. While integrating features definitely showed significant
improvement in the predictions, it did have a couple of issues. Sometimes, the
prediction was affected by the presence of noise in the background, resulting
in an erroneous prediction, and at other times, the predicted crack turned out
to be thicker than the actual, resulting in the loss of some fine details.
These shortcomings have been highlighted in the last 2 columns of 6.

These experiments demonstrate that adding prior information to the input
regarding the localization of cracks and scaling the network compute enough to
process the additional information leads to a subsequent increase in model
performance.

###  4.5 Comparative Analysis of Results

Figure 7: Comparison with state-of-the-art models Table 5:  Comparison with previous works evaluated on our dataset Technique | F1 Score | MIoU
---|---|---
VGG16 [49] | 0.3009 | 0.5444
DeepCrack [28] | 0.3404 | 0.5813
DAU-Net [34] | 0.5495 | 0.6857
DeepLab $\checkmark$ ResNet * (ours) | 0.7238 | 0.7712

We performed a systematic quantitative (Table 5) and qualitative (Fig. 7)
analysis of existing SOTA models for crack segmentation in the literature and
compared their performances to the proposed models. All the models were
evaluated on the test split of the refined dataset. We observed that DeepCrack
scales well to different sizes of cracks due to the aggregation of multi-scale
feature maps, but it does not accurately adapt to the varying thickness of
cracks and is prone to a lot of noise in the output. DAU-Net and VGG16 do not
perform well on untrained datasets; this highlights their inability to
generalize across different crack surfaces and illumination conditions, which
is largely attributed to the lack of diversity in their training data.
However, our model is not perfect. As seen in the diagram, the bad predictions
are marked with red circles. We observed that our model sometimes tended to
have predictions with small discontinuities and gaps, resulting in the loss of
crack information.

Analysis of results on different subsections of our dataset is presented in
Table 1 and Table 3; we can observe that even though our dataset contains
diverse crack images, the benchmarked models perform better for sub-datasets
Volker, DeepCrack and Crack500 because they contain a larger number of images.
Masonry and Ceramic datasets had very different background textures from the
other images, which caused a downgrade in the model’s performance. Analysis of
the performance of benchmarked models based on the spatial types of cracks
highlighted that, on average Linear cracks had the best predictions, followed
by branched cracks, and the webbed cracks had the lowest quality of
predictions.

##  5 Conclusion

Within the past decade, Computer Vision techniques for crack detection have
gained significant momentum and led to stimulating works and datasets.
However, most works focus on a particular dataset and do not benchmark the
results on other datasets, all while using varying metrics. This leads to
problems in reproducibility and/or limitation in application to different
types of cracks such as asphalt, concrete, masonry, ceramic and glass.

This paper tries to address the aforementioned vital issues that are
pertinent. We do so by compiling various datasets and putting together a
robust dataset consisting of 9000+ images with varying backgrounds, types of
cracks and surfaces, and ground truth annotations. Further, we refined
existing datasets and standardized the dataset by maintaining consistency in
the image resolution. We further divide the dataset on the basis of the
spatial arrangement of cracks into linear, branched and webbed. Finally, we
benchmarked the state-of-the-art techniques on our final dataset.

Our results show that DeepLab with ResNet and XceptionNet as backbone perform
the best. We further observe that the models perform best on images with
linear cracks, and predictions’ quality decreases for webbed and branched
cracks. The paper highlights the advantages of using self-supervised attention
in modelling long-range dependencies in vision tasks and presents a method to
fuse semi-supervised attention feature maps with CNNs for enhanced crack
detection. Adding the DINO features as prior results in significant
improvements in the predictions with a marginal tradeoff in the number of
trainable parameters.

Despite our best efforts in compiling a dataset containing cracks in different
domains, the models trained by us have not been able to identify cracks well
in the case of transparent backgrounds like glass. We believe in the need for
more images belonging to such backgrounds so that models trained on them shall
be more domain invariant. Further exploration into fusing semi-supervised and
supervised training paradigms and better amalgamation of transformer and CNN
architectures can be explored for better performance on images with branched
and webbed cracks.

## References

  * [1] Akagic, A., Buza, E., Omanovic, S., Karabegovic, A.: Pavement crack detection using otsu thresholding for image segmentation. In: 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). pp. 1092–1097. IEEE (2018)
  * [2] Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., Wang, M.: Swin-unet: Unet-like pure transformer for medical image segmentation (2021). https://doi.org/10.48550/ARXIV.2105.05537, https://arxiv.org/abs/2105.05537
  * [3] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers (2021). https://doi.org/10.48550/ARXIV.2104.14294, https://arxiv.org/abs/2104.14294
  * [4] Chambo, S.: Aiglern, https://www.irit.fr/Sylvie.%****␣038.bbl␣Line␣25␣****Chambon/Crack/Detection/Database.htm
  * [5] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs (2016). https://doi.org/10.48550/ARXIV.1606.00915, https://arxiv.org/abs/1606.00915
  * [6] Choi, W., Cha, Y.J.: Sddnet: Real-time crack segmentation. IEEE Transactions on Industrial Electronics 67(9), 8016–8025 (2019)
  * [7] Chollet, F.: Xception: Deep learning with depthwise separable convolutions (2016). https://doi.org/10.48550/ARXIV.1610.02357, https://arxiv.org/abs/1610.02357
  * [8] Dais, D., İhsan Engin Bal, Smyrou, E., Sarhosis, V.: Automatic crack classification and segmentation on masonry surfaces using convolutional neural networks and transfer learning. Automation in Construction 125, 103606 (2021). https://doi.org/https://doi.org/10.1016/j.autcon.2021.103606, https://www.sciencedirect.com/science/article/pii/S0926580521000571
  * [9] Dorafshan, S., Thomas, R., Maguire, M.: Sdnet2018: An annotated image dataset for non-contact concrete crack detection using deep convolutional neural networks. Data in Brief 21 (11 2018). https://doi.org/10.1016/j.dib.2018.11.015
  * [10] Eisenbach, M., Stricker, R., Seichter, D., Amende, K., Debes, K., Sesselmann, M., Ebersbach, D., Stoeckert, U., Gross, H.M.: How to get pavement distress detection ready for deep learning? a systematic approach. In: International Joint Conference on Neural Networks (IJCNN). pp. 2039–2047 (2017)
  * [11] Fan, R., Bocus, M.J., Zhu, Y., Jiao, J., Wang, L., Ma, F., Cheng, S., Liu, M.: Road crack detection using deep convolutional neural network and adaptive thresholding (2019)
  * [12] Flah, M., Suleiman, A.R., Nehdi, M.L.: Classification and quantification of cracks in concrete structures using deep learning image-based techniques. Cement and Concrete Composites 114, 103781 (2020). https://doi.org/https://doi.org/10.1016/j.cemconcomp.2020.103781, https://www.sciencedirect.com/science/article/pii/S0958946520302870
  * [13] Gao, Z., Peng, B., Li, T., Gou, C.: Generative adversarial networks for road crack image segmentation. In: 2019 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2019)
  * [14] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks (2014). https://doi.org/10.48550/ARXIV.1406.2661, https://arxiv.org/abs/1406.2661
  * [15] HamSangwoo, BaeSoohyeon, KimHwiyoung, LeeImpyeong, LeeGyu-Phil, KimDonggyou: Training a semantic segmentation model for cracks in the concrete lining of tunnel. Journal of Korean Tunnelling and Underground Space Association 23(6), 549–558 (11 2021)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. https://doi.org/10.48550/ARXIV.1512.03385, https://arxiv.org/abs/1512.03385
  * [17] Hoang, N.D.: Detection of surface crack in building structure using image processing technique with an improved otsu method for image thresholding. Advances in Civil Engineering 2018 (04 2018). https://doi.org/10.1155/2018/3924120
  * [18] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks (2018)
  * [19] Junior, G.S., Ferreira, J., Millán-Arias, C., Daniel, R., Junior, A.C., Fernandes, B.J.T.: Ceramic cracks segmentation with deep learning. Applied Sciences 11(13) (2021). https://doi.org/10.3390/app11136017, https://www.mdpi.com/2076-3417/11/13/6017
  * [20] Kalfarisi, R., Wu, Z., Soh, K.: Crack detection and segmentation using deep learning with 3d reality mesh model for quantitative assessment and integrated visualization. Journal of Computing in Civil Engineering 34, 04020010 (2020)
  * [21] Khalesi, S., Ahmadi, A.: Automatic road crack detection and classification using image processing techniques, machine learning and integrated models in urban areas: A novel image binarization technique (06 2020)
  * [22] Kheradmandi, N., Mehranfar, V.: A critical review and comparative study on image segmentation-based techniques for pavement crack detection. Construction and Building Materials 321, 126162 (2022). https://doi.org/https://doi.org/10.1016/j.conbuildmat.2021.126162, https://www.sciencedirect.com/science/article/pii/S0950061821038940
  * [23] Kim, B., Yuvaraj, N., Sri Preethaa, K., Arun Pandian, R.: Surface crack detection using deep learning with shallow cnn architecture for enhanced computation. Neural Computing and Applications 33(15), 9289–9305 (2021)
  * [24] König, J., Jenkins, M.D., Mannion, M., Barrie, P., Morison, G.: Optimized deep encoder-decoder methods for crack segmentation. Digital Signal Processing 108, 102907 (Jan 2021). https://doi.org/10.1016/j.dsp.2020.102907, http://dx.doi.org/10.1016/j.dsp.2020.102907
  * [25] Lau, S.L., Chong, E.K., Yang, X., Wang, X.: Automated pavement crack segmentation using u-net-based convolutional neural network. IEEE Access 8, 114892–114899 (2020)
  * [26] Lee, D., Kim, J., Lee, D.: Robust concrete crack detection using deep learning-based semantic segmentation. International Journal of Aeronautical and Space Sciences 20(1), 287–299 (2019)
  * [27] Li, S., Gu, X., Xu, X., Xu, D., Zhang, T., Liu, Z., Dong, Q.: Detection of concealed cracks from ground penetrating radar images based on deep learning algorithm. Construction and Building Materials 273, 121949 (2021)
  * [28] Liu, Y., Yao, J., Lu, X., Xie, R., Li, L.: Deepcrack: A deep hierarchical feature learning architecture for crack segmentation. Neurocomputing 338, 139–153 (2019)
  * [29] Mahler, D.S., Kharoufa, Z.B., Wong, E.K., Shaw, L.G.: Pavement distress analysis using image processing techniques. Computer-Aided Civil and Infrastructure Engineering 6(1), 1–14 (1991)
  * [30] Mandal, V., Uong, L., Adu-Gyamfi, Y.: Automated road crack detection using deep convolutional neural networks. In: 2018 IEEE International Conference on Big Data (Big Data). pp. 5212–5215. IEEE (2018)
  * [31] Oliveira, H., Correia, P.L.: Road surface crack detection: Improved segmentation with pixel-based refinement. In: 2017 25th European Signal Processing Conference (EUSIPCO). pp. 2026–2030. IEEE (2017)
  * [32] Pak, M., Kim, S.: Crack detection using fully convolutional network in wall-climbing robot. In: Park, J.J., Fong, S.J., Pan, Y., Sung, Y. (eds.) Advances in Computer Science and Ubiquitous Computing. pp. 267–272. Springer Singapore, Singapore (2021)
  * [33] Park, S.E., Eem, S.H., Jeon, H.: Concrete crack detection and quantification using deep learning and structured light. Construction and Building Materials 252, 119096 (2020)
  * [34] Polovnikov, V., Alekseev, D., Vinogradov, I., Lashkia, G.V.: Daunet: Deep augmented neural network for pavement crack segmentation. IEEE Access 9, 125714–125723 (2021). https://doi.org/10.1109/ACCESS.2021.3111223
  * [35] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 779–788 (2016). https://doi.org/10.1109/CVPR.2016.91
  * [36] Rezaie, A., Achanta, R., Godio, M., Beyer, K.: Comparison of crack segmentation using digital image correlation measurements and deep learning. Construction and Building Materials 261, 120474 (2020). https://doi.org/https://doi.org/10.1016/j.conbuildmat.2020.120474, https://www.sciencedirect.com/science/article/pii/S095006182032479X
  * [37] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation (2015). https://doi.org/10.48550/ARXIV.1505.04597, https://arxiv.org/abs/1505.04597
  * [38] Sharma, S., Balakrishnan, D., Kulkarni, S., Singh, S., Devunuri, S., Korlapati, S.C.R.: Crackseg9k: A Collection of Crack Segmentation Datasets (2022). https://doi.org/10.7910/DVN/EGIEBY, https://doi.org/10.7910/DVN/EGIEBY
  * [39] Shi, Y., Cui, L., Qi, Z., Meng, F., Chen, Z.: Automatic road crack detection using random structured forests. IEEE Transactions on Intelligent Transportation Systems 17(12), 3434–3445 (2016)
  * [40] Shim, S., Kim, J., Cho, G.C., Lee, S.W.: Multiscale and adversarial learning-based semi-supervised semantic segmentation approach for crack detection in concrete structures. IEEE Access 8, 170939–170950 (2020)
  * [41] Silva, W.R.L.d., Lucena, D.S.d.: Concrete cracks detection based on deep learning image classification. In: Proceedings. vol. 2, p. 489. MDPI AG (2018)
  * [42] Silva, W.R.L.d., Lucena, D.S.d.: Concrete cracks detection based on deep learning image classification. Proceedings 2(8) (2018). https://doi.org/10.3390/ICEM18-05387, https://www.mdpi.com/2504-3900/2/8/489
  * [43] Sonka, M., Hlavac, V., Boyle, R.: Segmentation, ch. 5 of image processing. Analysis and Machine Vision, PWS Publishing pp. 158–163 (1999)
  * [44] Sun, X., Xie, Y., Jiang, L., Cao, Y., Liu, B.: Dma-net: Deeplab with multi-scale attention for pavement crack segmentation. IEEE Transactions on Intelligent Transportation Systems pp. 1–12 (2022). https://doi.org/10.1109/TITS.2022.3158670
  * [45] Talab, A.M.A., Huang, Z., Xi, F., HaiMing, L.: Detection crack in image using otsu method and multiple filtering in image processing techniques. Optik 127(3), 1030–1033 (2016)
  * [46] Yamane, T., Chun, P.j.: Crack detection from a concrete surface image based on semantic segmentation using deep learning. Journal of Advanced Concrete Technology 18(9), 493–504 (2020)
  * [47] Yang, F., Zhang, L., Yu, S., Prokhorov, D.V., Mei, X., Ling, H.: Feature pyramid and hierarchical boosting network for pavement crack detection. IEEE Transactions on Intelligent Transportation Systems 21, 1525–1535 (2020)
  * [48] Yang, F., Zhang, L., Yu, S., Prokhorov, D., Mei, X., Ling, H.: Feature pyramid and hierarchical boosting network for pavement crack detection (2019)
  * [49] Zhang, L., Shen, J., Zhu, B.: A research on an improved unet-based concrete crack detection algorithm. Structural Health Monitoring 20(4), 1864–1879 (2021). https://doi.org/10.1177/1475921720940068, https://doi.org/10.1177/1475921720940068
  * [50] Zhang, X., Rajan, D., Story, B.: Concrete crack detection using context-aware deep semantic segmentation network. Computer-Aided Civil and Infrastructure Engineering 34(11), 951–971 (2019)
  * [51] Zou, Q., Cao, Y., Li, Q., Mao, Q., Wang, S.: Cracktree: Automatic crack detection from pavement images. Pattern Recognition Letters 33(3), 227–238 (2012)
  * [52] Özgenel, ., Sorguc, A.: Performance comparison of pretrained convolutional neural networks on crack detection in buildings (07 2018). https://doi.org/10.22260/ISARC2018/0094

##  Appendix 0.A Experimental Settings

For all our experiments, we use a stratified train test split of 90-10% on the
compiled dataset. All the benchmarked models have been trained on Nvidia-
GeForce 1080 Ti with 12 GB of Memory.Early Stopping with a patience of 5
epochs was used to determine convergence for all the models.

Table 6: Model Hyperparameters for reproducibility Model | Optimizer & | Epochs | Pre-Trained | Encoder
---|---|---|---|---
| Learning Rate |  |  |
MaskRCNN | Adam , 1e-4 | 20 | ImageNet | Resnet-50
Swin-Transformer | Adam , 1e-1 | 20 | ImageNet | Swin-Unet
DeepLab Resnet | Adam , 7e-2 | 25 | ImageNet | Resnet-101
DeepLab Xception | Adam , 7e-2 | 25 | ImageNet | Xceeption
Pix2Pix U-Net | Adam , 1e-3 | 30 | ImageNet | VGG-16
Pix2Pix FPN | Adam , 1e-3 | 30 | ImageNet | Resnet-50

##  Appendix 0.B Dataset details

The dataset published here is the largest, most diverse and consistent crack
segmentation dataset constructed so far. It contains 9255 images that combine
different smaller open source datasets. It consists of 10 sub datasets
preprocessed and resized to 400x400 namely, Crack500, Deepcrack, Sdnet,
Cracktree, Gaps, Volker, Rissbilder, Noncrack, Masonry and Ceramic.
The dataset has been made publicly available and can be found at Harvard
Database: https://doi.org/10.7910/DVN/EGIEBY.

##  Appendix 0.C Deployment of Segmentation Pipeline using Stitching

Inference on images with resolution higher than the input size of the model
can be obtained by interpolation/ resizing but this leads to sub-par
performance and artifacts in the predictions. Here, we propose an improved
technique of ‘stitching’ together the inference on crops of smaller size to
produce the final result. In this technique, we divide the image into multiple
subimages such that their size is equal to that of the size that the deep
learning network has been trained on. Later the predictions are combined such
that their spatial locations are consistent to the original input image. Using
this technique enables us to by-pass the need of YOLO for detecting crops and
resizing them. The qualitative comparison of ‘stitching’ and the previous
pipeline is presented in Fig. 8.

Figure 8: Transformation in Predictions due to Stitching. Model Used: Pix2Pix
FPN