# Multimodal Deep Learning Approach to Predicting
Neurological Recovery From Coma After Cardiac Arrest

Felix H. Krones1 Ben Walker2 Guy Parsons1 Terry Lyons2 Adam Mahdi1

1 Oxford Internet Institute  University of Oxford  Oxford  UK
2 Mathematical Institute  University of Oxford  Oxford  UK

###### Abstract

This work showcases our team’s (The BEEGees) contributions to the 2023 George
B. Moody PhysioNet Challenge. The aim was to predict neurological recovery
from coma following cardiac arrest using clinical data and time-series such as
multi-channel EEG and ECG signals. Our modelling approach is multimodal, based
on two-dimensional spectrogram representations derived from numerous EEG
channels, alongside the integration of clinical data and features extracted
directly from EEG recordings. Our submitted model achieved a Challenge score
of $0.53$ on the hidden test set for predictions made $72$ hours after return
of spontaneous circulation. Our study shows the efficacy and limitations of
employing transfer learning in medical classification. With regard to
prospective implementation, our analysis reveals that the performance of the
model is strongly linked to the selection of a decision threshold and exhibits
strong variability across data splits.

##  1 Introduction

There are more than six million cardiac arrests annually, with general
survival rates varying between $1\%$ and $10\%$ due to geographical
disparities [1]. Following successful resuscitation, a significant proportion
of survivors are admitted to intensive care units (ICUs) in a comatose state,
with severe brain injury emerging as the leading cause of death among this
group. In the crucial days following cardiac arrest, medical professionals are
often tasked with estimating the likelihood of the patient regaining
consciousness. A positive prognosis often leads to ongoing medical care, while
a negative prognosis can result in the discontinuation of life support and
hence death. There have been reported instances of patients recovering well
despite a grim prognosis, raising concerns that negative predictions may
inadvertently influence the outcome [2].

This year’s Challenge [1, 3] asked to develop an open-source algorithm capable
of predicting the extent of recovery from coma after a cardiac arrest. These
predictions were to be made using a combination of basic clinical data, EEG,
ECG and other signals, with the aim of classifying outcomes into either ’Poor’
or ’Good’. In this work we develop a multimodal deep learning approach. Our
strategy involves generating two-dimensional spectrogram representations
sourced from multi-channel EEG signals and their integration with clinical
data, along with features directly extracted from the EEG recordings.

##  2 Methodology

###  2.1 Data

The dataset for this study is taken from the International Cardiac Arrest
REsearch (I-CARE) consortium and originates from seven academic hospitals
across the United States and Europe [4]. It comprises clinical data, including
age, gender, return time of spontaneous circulation (ROSC), arrest location
(out-of-hospital cardiac arrest OHCA), presence of shockable rhythm, use of
targeted temperature management (TTM), and clinical time-series data, such as
continuous electroencephalography (EEG), electrocardiogram (ECG) and partially
other recordings (e.g. SpO2). The dataset consists of $1,\\!020$ patients
(from which $607$ were provided for training) who were admitted to an ICU in a
comatose state following cardiac arrest. Neurological outcomes were assessed
using the Cerebral Performance Category (CPC) scale.

We filtered the EEG signals by applying a band-pass filter over the range
$0.5$–$30$ Hz and a notch-filter at $50$ and $60$ Hz to mitigate artifacts
from electrical grids. We employed artifact detection using a sliding window
approach, only keeping the cleanest section of each recording.

For part of our model (Sec. 2.2) we converted the EEG signals to spectrograms
(Figure 1), using the Python package librosa, to use them together with other
non-imaging modalities as inputs for the models [5].

Figure 1: An example of EEG recordings (bottom) for patient 994 at the 8th
hour of channel F8 with good outcome. The top displays the corresponding
spectrogram (squared amplitude in decibel units relative to peak power).

###  2.2 Models and architecture

In our approach, outlined in Figure 2, we evaluated six models for binary
prediction of patient outcome at 72 hours after ROSC. Model M1 used clinical
features and EEG summary statistics before employing a Random Forest
classifier. In model M2, we added features extracted from EEG spectrograms
using a DenseNet121-CNN [6] into the classifier. In model M3, these DenseNet
features were further aggregated over time and channels. Model M4 introduced
an intermediate fusion step for clinical features into the last layer of the
DenseNet architecture. In models M5 and M6, we introduce additional output
from a ridge regression classifier. This classifier is trained on features
extracted from the EEG signals using the Random Convolutional Kernel Transform
(ROCKET) [7]. We obtain the regularisation strength of the classifier through
leave-one-out cross-validation over $10$ log-evenly spaced values ranging from
$10^{-3}$ and $10^{3}$. ROCKET employs $10^{4}$ kernels, with lengths randomly
selected from the set $\\{7,9,11\\}$. Each kernel comprises $4$ features and a
maximum of $32$ dilations. These settings align with the default parameters
from sktime’s RocketClassifier. Model M5 omitted the intermediate fusion
present in model M4, whereas model M6 included it.

Figure 2: A schematic diagram of our model architecture. Blue: input data;
red: filters and aggregation; yellow: pre-defined features; green: trainable
models.

###  2.3 Scoring

The Challenge’s official score emphasises the False Positive Rate (FPR) of
incorrectly predicting a poor patient outcome. The scoring method selects the
highest decision threshold that maintains the FPR below $5\%$, and
subsequently evaluates the True Positive Rate (TPR) for predicting poor
outcomes. Mathematically the score is defined, given a threshold $\theta$
drawn from the predictions, as

$\max_{\theta:\mathrm{FPR}_{\theta}\leq 0.05}\mathrm{TPR}_{\theta}$ (1)

Two types of scores were reported. The ‘Validation Score’ was the assessment
provided by the Challenge organisers on the hidden set. The ‘CV Score’ and ‘CV
AUC’ were computed as the mean and standard deviation, derived from the local
five-fold cross-validation (CV).

##  3 Results

Table 1 shows the feature availability and their overall key characteristics
for the training data provided by the organisers of the Challenge collected
from $607$ patients.

Feature | Value |  Missing
---|---|---
Age [years], mean (SD) | 61 (16) | 1
Sex [‘Men’], N(%) | 417 (69) | 0
ROSC [minutes], mean (SD) | 23 (19) | 304
OHCA [‘True’], N(%) | 442 (78) | 41
Shockable rhythm [‘True’], N(%) | 297 (52) | 32
TTM, N at 33/36/na °C | 448/61/98 | 0
Outcome [‘Poor’], N(%) | 382 (52) | 0
Table 1: Summary of clinical data and patient outcome for the available $607$
patients. OHCA is ‘True’ for out of hospital arrests; TTM is the temperature
management where ‘na’ stands for no TTM was applied.

Table 2 provides the official scores on the hidden validation set, along with
the mean and standard deviation for the score and AUC obtained from our local
cross-validation, for the six models proposed in this work. Our final
submission (model M5) achieved a Challenge score of $0.53$ on the hidden test
set.

Model | Features | Validation Score | CV Score | CV AUC
---|---|---|---|---
M1 | Clinical data, EEG summary stats, Signal flags | 0.520 | 0.327 (0.137) | 0.688 (0.046)
M2 | \+ DenseNet on spectrograms | 0.540 | 0.484 (0.157) | 0.824 (0.023)
M3 | \+ aggregation of features over time and channels | 0.567 | 0.527 (0.090) | 0.811 (0.049)
M4 | \+ intermediate fusion | 0.328 | 0.537 (0.104) | 0.818 (0.038)
M5 | \+ ROCKET features without intermediate fusion | 0.627 | 0.447 (0.085) | 0.836 (0.033)
M6 | \+ ROCKET features with intermediate fusion | not provided | 0.567 (0.085) | 0.854 (0.015)
Table 2: The six models proposed and tested in this work (Sec. 2.2). The
Validation Score is the performance on the Challenge’s hidden validation set.
CV Score is the mean (standard deviation) Challenge score from the five-fold
cross validation on the training data (Sec. 2.3). CV AUC is the AUC on the
same data. The ‘$+$’ should be read as ‘in addition’.

Figure 3 shows the relative feature importance of the classifier of our
locally best performing model M6. The EEG signal information were the most
important features while demographics and other clinical features exhibit
lower importance. Figure 4 shows the dependency of error rates on the
selection of a particular threshold. Figure 5 shows the ROC curve of our
locally best performing approach (model M6) for which AUC=$0.854$.

Figure 3: Relative feature importance plot for the locally best performing
model M6. Here ‘agg.’ means aggregated over channels and time using the mean
prediction and a majority voting for the DenseNet features. Figure 4: Accuracy
(blue), FPR (red dashed) and FNR (red dashed-dotted) of the ‘Poor’ outcome
label for different decision thresholds of model M6. Figure 5: ROC curve (blue
solid) of our final model (AUC = 0.854) with the indicated (black line) 5% FPR
threshold.

##  4 Discussion

###  4.1 Main findings and limitations

In this study, we developed and tested six models for forecasting post-cardiac
arrest coma recovery using a multimodal approach [8, 5, 9]. Model M1 served as
our benchmark. Incorporating CNN-extracted features from spectrograms in M2
led to the most significant AUC improvement in local CV. To improve
robustness, we aggregated features from different hours and channels in model
M3. This aggregation effectively reduced the number of features used in the
final classifier and improved the score, which was highly sensitive to minor
variations on the left side of the AUC curve (Figure 5). Although model M4’s
intermediate fusion improved local score, its validation performance declined,
indicating overfitting. Adding features extracted by ROCKET to the final
classifier substantially improved performance on the validation set (0.627),
but had limited impact on the local CV set. Furthermore, performance
deteriorated on the hidden test set (0.53), leading us to conclude that
enhancing the robustness of our methodology remains an open research question.
Additionally, given that ROCKET features did not markedly improve local
results, it is still unclear how much additional information these features
provide compared to those already extracted. Our results show that although
multimodal approaches (M2-M6) achieved strong performance on the local held-
out set (${\rm CV\,\,AUC}>0.81$), generalisability concerns emerged, which
need further considerations.

###  4.2 Previous work

The majority of the predictive models developed to date using EEG signals
predominantly employ CNNs, using colour channels for diverse signals. Notably,
these models are mostly constructed from datasets consisting of fewer than
$300$ patients [10]. Only a handful of studies have focused on predicting coma
outcomes [10, 11, 12]. For example, [11] achieved an AUC of 91% at 66 hours
after return of spontaneous circulation with a sensitivity of 66% for poor
outcome prediction at a specificity of 95%.

###  4.3 Future research

Several potential avenues for future research are available. For instance, one
can compare different modelling approaches [13, 11] and exploring self-
supervised pre-training to boost network performance. The incorporation of
other time-series data, such as ECG (not fully exploited in this study due to
substantial missing data) could also lead to valuable insights, particularly
when integrated with EEG signals [14]. This year’s Challenge, however,
demonstrated that the best-performing and most robust models were those that
had undergone substantial feature engineering and data pre-processing [14].
The winning team, for example, extracted 362 expert-based EEG features plus
additional ECG features [14].

## Code availability

Our complete code is available on GitHub at
https://github.com/felixkrones/physionet_challenge_2023.

## Acknowledgement

FK was partially supported by the Friedrich Naumann Foundation. BW and TL were
partially funded by the Hong Kong Innovation and Technology Commission. TL was
funded in part by the EPSRC [EP/S026347/1], The Alan Turing Institute
[EP/N510129/1], the Data Centric Engineering Programme (Lloyd’s Register
Foundation G0095), the Defence and Security Programme (UK Government funded),
and the Office for National Statistics. Funding bodies had no influence on the
research. No conflicts of interest existed.

## References

  * [1] Reyna MA, Amorim E, Sameni R, Weigle J, Elola A, Bahrami A, Seyedi S, Kwon H, Zheng WL, Ghassemi M, van Putten MJAM, Hofmeijer J, Gaspard N, Sivaraju A, Herman S, Lee JW, Westover BM, Clifford GD.  Predicting Neurological Recovery from Coma After Cardiac Arrest: The George B. Moody PhysioNet Challenge 2023.  Computing in Cardiology 2023;50:1–4.
  * [2] Forgacs PB, Devinsky O, Schiff ND.  Independent functional outcomes after prolonged coma following cardiac arrest: a mechanistic hypothesis.  Annals of Neurology 2020;87(4):618–632.
  * [3] Goldberger AL, Amaral LA, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng CK, Stanley HE.  PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals.  Circulation 2000;101(23):e215–e220.
  * [4] Amorim E, Zheng WL, Ghassemi MM, Aghaeeaval M, Kandhare P, Karukonda V, Lee JW, Herman ST, Sivaraju A, Gaspard N, Hofmeijer J, van Putten MJAM, Sameni R, Reyna MA, Clifford GD, Westover MB.  The International Cardiac Arrest Research (I-CARE) Consortium Electroencephalography Database.  In Critical Care Medicine 2023; (in press).
  * [5] Walker B, Krones F, Kiskin I, Parsons G, Lyons T, Mahdi A.  Dual Bayesian ResNet: A deep learning approach to heart murmur detection.  In 2022 Computing in Cardiology (CinC). 2022; 1–4.
  * [6] Huang G, Liu Z, Van Der Maaten L, Weinberger KQ.  Densely connected convolutional networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017; 4700–4708.
  * [7] Dempster A, Petitjean F, Webb GI.  ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels.  Data Mining and Knowledge Discovery 2020;34:1454–1495.
  * [8] Grant D, Papież B, Parsons G, Tarassenko L, Mahdi A.  Deep learning classification of cardiomegaly using combined imaging and non-imaging ICU data.  In Medical Image Understanding and Analysis. Springer, 2021; 547–558.
  * [9] Duvieusart B, Krones F, Parsons G, Tarassenko L, Papież B, Mahdi A.  Multimodal Cardiomegaly Classification with Image-Derived Digital Biomarkers.  In Medical Image Understanding and Analysis. 2022; 13–27.
  * [10] Jonas S, Rossetti AO, Oddo M, Jenni S, Favaro P, Zubler F.  EEG-based outcome prediction after cardiac arrest with convolutional neural networks: performance and visualization of discriminative features.  Human Brain Mapping 2019;40(16):4606–4617.
  * [11] Zheng WL, Amorim E, Jing J, Ge W, Hong S, Wu O, Ghassemi M, Lee JW, Sivaraju A, Pang T, Herman ST, Gaspard N, Ruijter BJ, Sun J, Tjepkema-Cloostermans MC, Hofmeijer J, van Putten MJ, Westover MB.  Predicting neurological outcome in comatose patients after cardiac arrest with multiscale deep neural networks.  Resuscitation 2021;169:86–94.
  * [12] Tjepkema-Cloostermans MC, da Silva Lourenço C, Ruijter BJ, Tromp SC, Drost G, Kornips FH, Beishuizen A, Bosch FH, Hofmeijer J, van Putten MJ.  Outcome prediction in postanoxic coma with deep learning.  Critical Care Medicine 2019;47(10):1424–1432.
  * [13] Andreotti F, Carr O, Pimentel M, Mahdi A, De Vos M.  Comparing feature-based classifiers and convolutional neural networks to detect arrhythmia from short segments of ECG.  In 2017 Computing in Cardiology (CinC). 2017; 1–4.
  * [14] Zabihi M, Zar AC, Grover P, Rosenthal ES.  HyperEnsemble learning from multimodal biosignals to robustly predict functional outcome after cardiac arrest.  2023 Computing in Cardiology 2023;1–4.


#  NHA12D: A NEW PAVEMENT CRACK DATASET AND A COMPARISON STUDY OF CRACK
DETECTION ALGORITHMS

Zhening Huang1, Weiwei Chen1, Abir Al-Tabbaa1, Ioannis Brilakis1
1Department of Engineering, University of Cambridge

## Abstract

Crack detection plays a key role in automated pavement inspection. Although a
large number of algorithms have been developed in recent years to further
boost performance, there are still remaining challenges in practice, due to
the complexity of pavement images. To further accelerate the development and
identify the remaining challenges, this paper conducts a comparison study to
evaluate the performance of the state of the art crack detection algorithms
quantitatively and objectively. A more comprehensive annotated pavement crack
dataset (NHA12D) that contains images with different viewpoints and pavements
types is proposed. In the comparison study, crack detection algorithms were
trained equally on the largest public crack dataset collected and evaluated on
the proposed dataset (NHA12D). Overall, the U-Net model with VGG-16 as
backbone has the best all-around performance, but models generally fail to
distinguish cracks from concrete joints, leading to a high false-positive
rate. It also found that detecting cracks from concrete pavement images still
has huge room for improvement. Dataset for concrete pavement images is also
missing in the literature. Future directions in this area include filling the
gap for concrete pavement images and using domain adaptation techniques to
enhance the detection results on unseen datasets.

## Introduction

Automated pavement inspection involves taking RGB images of pavements with
surveying vehicles and assessing their condition with image processing
technologies. Crack detection is the task to extract crack information from 2D
pavement images. There are various definitions of crack detection, and this
paper focuses on pixel-level crack detection, which classifies each pixel in
an image into the crack or non-crack pixel. Cracking is the most common type
of surface distress on pavements and the severity of cracking on pavements is
an essential indicator of pavement conditions (Ragnoli, De Blasiis and Di
Benedetto (2018)). As a result, crack detection plays a key role in pavement
management systems for determining optimal maintenance strategies. Crack
detection from 2D images is a long-standing research challenge due to its
inherent irregular patterns of crack, noise in images and different lighting
conditions. In the early days, researchers used image processing techniques
such as edge detection to extract the skeleton of cracks from images. In
recent years, a large number of machine learning-based crack detection
algorithms, such as DeepCrack ( Jenkins et al. (2018)), Crackforest(Shi et al.
(2016)) have been developed and many of them demonstrated promising
performance. However, to date, fully automatic crack detection remains a
challenge as in practice, pavement images captured on highways have more
complex backgrounds and conditions than images in public datasets.
To further accelerate the performance, we propose a new dataset, which
contains images with more comprehensive cases, such as different pavement
types and viewpoints. The pavement images in the proposed dataset were
captured by National Highway’s digital survey vehicles on the A12 network, the
UK. Cracks in the proposed dataset were visually identified and labelled at
the pixel level. Moreover, a benchmark study for the state of the art (SOTA)
crack detection algorithms is conducted, where three SOTA crack detection
algorithms were selected and tested on the proposed dataset. The performance
of three different detection models is quantitatively and objectively
evaluated with consistent evaluation matrices. The detection results, as an
example shown in Figure 1, are used to identify the remaining challenges and
potential directions for future development.
This paper is organized as follows. The section Introduction briefly
introduces the background, research gap, and objectives for this study. The
section “Related Work” describes the state of the art crack detection
algorithms, existing benchmark studies and available pixel-level crack
detection datasets. The section “Methodology” discusses the proposed dataset
and implementation details for the comparison study. The section “Results and
Discussion” evaluates the numerical and visual results of the models and
discusses outcomes. The potential future research direction is also
highlighted. The section “Conclusions” presents the study conclusions and
highlights recommendations for future research.

Figure 1: Crack detection results on the proposed dataset. SoTA crack
detection algorithms still have outstanding high false positive rates on
concrete pavement images.
Black pixels: true positive; red pixels: false positive; green pixels: false
negative

## Related Work

In the past two decades, methods to detect cracks from 2D pavement images have
been investigated extensively as the need to achieve automated pavement
inspection arises. In general, methods proposed by different scholars can be
divided into two categories: Image processing-based methods and machine
learning-based methods. Image processing-based methods use techniques such as
intensity thresholding (Wang and Tang (2011); Katakam (2009); Oliveira and
Correia (2009); Li et al. (2016); Tang and Gu (2013); Hu and Zhao (2010)),
edge detection (Zhao, Qin and Wang (2010); Attoh-Okine and Ayenu-Prah (2008);
Wang, Li and Gong (2007); Yan et al. (2007)) to extract crack skeleton from
images. However, these methods tend to fail under complicated backgrounds and
have low robustness against the change of environments. Machine learning-based
methods, especially deep learning methods, although require a large amount of
annotated datasets for training, have shown impressive results in recent years
and outperform image processing-based methods in many dimension.

### Crack Detection Algorithms

In the past five years (2016-2021), numerous machine learning algorithms were
developed for crack detection. Shi et al. (2016) proposed a novel crack
detection framework based on random structured forests named CrackForest. Some
general algorithms for 2D image segmentation, such as UNet (Jenkins et al.
(2018)), DeepLab (Chen and Jahanshahi (2018)) have also been proven to be very
powerful for crack segmentation. Lau et al. (2020) proposed a UNet based
network architecture in which they replace the encoder with a pre-trained
ResNet-34 neural network. Likewise, Dung and Anh (2019) proposed a method
based on a deep fully convolutional network (FCN) to detect cracks in concrete
images. Jenkins et al. (2018) proposed DeepCrack CNN for automatic crack
detection by learning high-level features for crack representation. Yang et
al. (2020) proposed a novel network architecture, named Feature Pyramid and
Hierarchical Boosting Network (FPHBN), for pavement crack detection. These
methods have all been tested on public data and proved to have great results.
However, most algorithms are only tested on one or two public datasets, while
their robustness across multiple datasets remains unevaluated.

### Comparison Studies

With the rapid emergence of new crack detection algorithms, the necessity to
benchmark different crack detection algorithms also increases. In the
literature, most model comparisons were conducted when a new algorithm was
proposed (Yang et al. (2020); Zou et al. (2019); Lau et al. (2020); Li et al.
(2019)). This type of comparison, however, tends to be inaccurate due to
several reasons. For example, in some publications, different CNN-based models
were evaluated without being trained on the same dataset, which makes the
comparison unrigorous as the training scheme has sufficient impacts on the
model performance. Also, the detection results in some papers were cited
directly from other papers without considering the image pre-processing steps,
implementation methods and evaluation details. Results from these comparisons
tend to subject to bias to support the proposed algorithms. In the past, there
are also some research work that specifically focused on model comparison. For
example, Tsai, Kaul and Mersereau (2010) carried out a benchmark study for
various crack detection models. In this work, six different algorithms were
tested on pavement images captured on interstate highway I-75/I-85 near
Atlanta and provided by the Georgia Department of Transportation. However, as
this work was conducted in 2010, before the rise of machine learning-based
methods, only image processing models were assessed. Dorafshan, Thomas and
Maguire (2018) also tested six edge detection models and three deep learning
models on concrete images. However, models used in that study detect cracks on
classification level and cannot achieve pixel-level detection. Hsieh and Tsai
(2020) carried out a benchmark for machine learning-based crack detection
models, but their focuses were on 3D crack pavement images that were captured
by laser scanners. Therefore, a comprehensive and robust comparison of the
latest 2D crack detection algorithms is still missing in the literature.

### Crack Detection Dataset

To fulfil the needs of benchmark study and training machine learning models,
several crack detection datasets were proposed in the literature. A
comprehensive search of the public crack dataset that labelled crack at the
pixel level is conducted and the available datasets are listed in Table 1. CFD
dataset (Shi et al. (2016)) is the most widely used dataset for crack
detection, which consists of 118 images of cracks on urban road surfaces in
Beijing taken by iPhone 5. Each image is resized to $480\times 320$ pixels and
has been labelled at the pixel level. Similarly, Crack500 (Lau et al. (2020))
that were collected on the main campus of Temple University using cell phones,
consists of 500 RGB images of pavement cracks of size around $2560\times 1440$
pixels that. The other seven datasets in the literature, along with their
parameters, are listed in table 1. In summary, most datasets were captured on
asphalt pavements, and images on concrete pavement is rare. Despite a large
number of available datasets in the literature, the dataset that covers a
range of complicated scenes on pavements is still needed.

Table 1: A comparison of existing crack dataset and the proposed NHA12D
dataset

[t]  Name Reference Resolution Type of Label Viewpoint Surface CFD1 Shi et al.
(2016) $480\times 320$ Pixel-Level Vertical Asphalt Crack500 Lau et al. (2020)
$2560\times 1440$ Pixel-Level Random Asphalt CrackTree260 Zou et al. (2012)
$960\times 720$ Skeleton2 Vertical Asphalt DeepCrack Zou et al. (2019)
$544\times 384$ Pixel-Level Random3 Random GAPs384 Yang et al. (2020)
$1920\times 1080$ Patch-Level4 Vertical Asphalt Stone331 Zou et al. (2019)
$1024\times 1024$ Pixel-Level Random Stone CRKWH100 Zou et al. (2019)
$512\times 512$ Pixel-Level Vertical Asphalt CrackLS315 Zou et al. (2019)
$512\times 512$ Pixel-Level Vertical Asphalt AigleRN S. Chambon (2011)
$991\times 462$ Pixel-Level Vertical Asphalt NHA12D – $1920\times 1080$ Pixel-
Level Vertical+Forwarding Concrete+Asphalt

  * 1

Most commonly used crack dataset to verify the detection results for crack
detection models

  * 2

CrackTree260 labels only depict the skeleton of the crack without rendering
the width

  * 3

DeepCrack contains a variety of surface including pavements, building walls,
stone, metals and so on.

  * 4

GAPs384 are divided into $64\times 64$ patches and each patch is labelled as a
crack or not

## Methodologies

### Proposed Dataset

An annotated road crack dataset named NHA12D is proposed. The proposed dataset
is directly taken by National Highways Surveying vehicles. This dataset is
composed of 80 pavement images, including 40 concrete pavement images and 40
asphalt pavement images. Each image has a resolution of $1920\times 1080$ and
captures the view for an individual lane on the A12. Some images also contain
vehicles, road markings, road studs, water stains, and other random objects.
Concrete pavements have transversal and longitudinal joints every certain
meter, which can be challenging for crack detection models to differentiate.
For different types of pavements, 25 images were taken by cameras with a
vertical viewpoint, and the other 15 were taken with a forwarding
camera111Available at https://github.com/ZheningHuang/NHA12D-Crack-Detection-
Dataset-and-Comparison-Study.

Each image was carefully labelled at the pixel level. The segmentation masks
are binary images where crack pixels have 255 as their intensity value while
the other areas, i.e. backgrounds have intensity values of 0, as shown in
Figure 2.

Figure 2: Our proposed Crack Detection Dataset (NHA12D) contains different
pavement types and viewpoints images. Each image captures the view of entire
lanes and provides the high-quality pixel-level crack annotation.
C: Concrete, A: Asphalt, F: Forwarding view, V: Vertical view

### Models selection

Three crack detection models were selected for this benchmark study,
including:

  * •

VGG-16/UNet (Dung and Anh (2019)), a model that uses VGG-16 as the backbone
encoder, UNet as the main architecture, and binary cross entropy as loss
function

  * •

DeepCrack (Zou et al. (2019)), a CNN-based model that is designed specifically
for crack segmentation

  * •

ResNet-34/UNet (Lau et al. (2020)), a model that has ResNet-34 as its
backbone, UNet as main architecture, and the model is trained directly by
optimizing the dice coefficient

## Implementation Details

All training experiments in this project were performed on Intel Core i7-8700
Six-Core CPU and a single NVIDIA Tesla P100 16 GB GPU. The implementation
details for different models are listed below:

  * •

VGG-16/UNet: The VGG-16/UNet architecture is built with implemented with
Pytorch API. The weights of VGG-16 pre-trained on the ImageNet dataset is used
for initialization.

  * •

DeepCrack: The DeepCrack Model was also developed with Pytorch. The source
code was published along with the paper and was used in this project for
training and testing (Zou et al. (2019)).

  * •

ResNet-34/UNet: A model to implement ResNet-34/UNet was developed with
Tensorflow/ Keras. The ResNet-34 encoder was pre-trained on ImageNet datasets.

### Training Scheme

To optimize the performance of learning based models, we trained all models on
a comprehensive dataset that contains all available crack datasets listed in
table 1. We also added some non-crack images for data balance purposes.
Overall, this merged dataset contains 11,220 images, with the data
distribution and pixel evaluation shown in Table 2. All images were resized to
$448\times 448$.

Table 2: Distribution of crack and Non-crack images in the merged dataset |  |
---|---|---
Image Type |  Number in Training Set |  Number in Validation Set
|  |
|  |
Crack Images |  8405 |  1484
|  |
|  |
Non-Crack Images |  1199 |  212

#### Data Augmentation

Image augmentation artificially creates training images through different
processing methods or a combination of multiple processing methods. It can
enlarge the training dataset and therefore boost the performance of models. In
this experiment, image augmentation was applied randomly to each image during
the training process to increase the number of training examples virtually.
Three types of augmentation were performed:

  * •

Rotate image randomly between 0∘ to 360 ∘

  * •

Random flips in horizontal and vertical axes

  * •

Random change in lighting

The same augmentation was also applied to the ground truth of each training
image, except for the random change in lighting.

### Testing Scheme

#### Patchwise testing scheme

Four crack detection model takes images input of size $256\times 256$. For
large images that have a size of $1920\times 1080$. This resizing process can
significantly reduce the information stored in the images and therefore harm
the testing results. To overcome this problem, we develop a new testing scheme
that first crops the raw images into 24 small patches, which have sizes of
$270\times 320$. Each patch was then tested independently with the detection
models and the testing results were merged to form the final prediction.
Another reason for cropping raw images is to compensate for the pixel ratio
difference between testing and training datasets. As shown in table 3, for
crack images in the training dataset, the crack pixel ratio is 4.66%, while in
the NHA12D dataset, this ratio is 0.40% originally. Cropping images using the
aforementioned method can significantly increase the ratio of crack pixels in
crack images by around six times, to 2.40%. Narrowing down the ratio
difference between testing and training datasets has the potentials to boost
the model performance.

Table 3: The percentage of crack and non-crack pixels in the NHA12D dataset, before and after the patch-wise pre-processing step |  |
---|---|---
Pre-processing |  Crack Pixel |  Non-crack Pixel
|  |
|  |
|  |
Before |  0.40% |  99.60%
|  |
|  |
After |  2.40% |  96.60%
|  |
|  |
Training Set |  4.66% |  95.34%

#### Otsu thresholding

The testing results from different models are greyscaled images that have
pixel intensity varies from 0 to 255. The value of each pixel represents the
likelihood of that pixel being a crack. However, the ground truth dataset is
made up of binary images with the crack pixels being 255 and non-crack pixels
being 0. To make them comparable, the Otsu thresholding method was applied to
binarize the testing results. Otsu threshold returns a single intensity
threshold that separates pixels into two classes, foreground and background,
by minimizing intra-class intensity variance. An example of this operation is
shown in Figure 3.

Figure 3: An illustration of using the Otsu threshold to binarize testing
results. Figure 4: The overall training and testing process of this benchmark
study. The NHA12D were split into patches before being tested on three CNN-
based models that were pre-trained on the merged dataset and the Localized
Thresholding model. The tested patches were merged to form the final testing
results.

#### Evaluation metrics

Precision, Recall and F1 score are the three most commonly used parameters for
crack detection evaluation, so we choose this metrics for this study. Three
parameters can be obtained by using the following equations.

$Precision=\frac{TP}{TP+FP}\qquad Recall=\frac{TP}{TP+FN}$
$F_{1}=\frac{2\times TP}{2\times TP+FP+FN}$

In the evaluation step, the value of TN, FP, TN, TP were calculated by
comparing the binary testing results of images to their ground truth
segmentation masks. Also because there are transition regions between the
crack pixels and the non-crack pixels in the subjectively labelled ground
truth, results that are two pixels near any labelled crack pixel are
considered as true positives. This evaluation convention is widely used in
other crack detection research ( Lau et al. (2020); Amhaz et al. (2016); Fan
et al. (2018); Liu et al. (2019); Ai et al. (2018)). The overall training and
testing process are illustrated in Figure 4.

## Results and Discussion

Numerical Results
The testing results of three models on different images were evaluated from
various perspectives. All results are summarized in Table 4. From the result,
the VGG-16/UNet model has the best overall performance, as the F1 score
indicates. The second-best performance model is the ResNet34/UNet model. From
The recall perspective, the DeepCrack model outperformed other models with a
very high margin, while from the precision perspective, two UNet-based models
demonstrated similar performance and outperform better than the other two
models. Models tested on concrete pavement images generally had higher recall
scores, but lower precision scores, due to the false positive results on
concrete joint region.

Figure 5: Eight examples of crack detection results by different models,
including 2 images for each pavement type on each viewpoint in NHA12D dataset.
Black pixels: True Positive; Red pixels: False Positive, Green Pixels: False
Negative Table 4: Quantitative comparisons of Four Algorithms on the NHA12D
dataset, with the highest value highlighted in bold

Pavement Type | Technique |  Vertical View | Forwarding View
---|---|---|---
Recall | Precision | F1-Score | Recall | Precision | F1-Score
Concrete | DeepCrack | 0.963 | 0.219 | 0.349 | 0.966 | 0.141 | 0.243
VGG16/UNet | 0.915 | 0.507 | 0.638 | 0.924 | 0.353 | 0.504
ResNet34/UNet | 0.935 | 0.427 | 0.570 | 0.922 | 0.250 | 0.385
Asphalt | DeepCrack | 0.903 | 0.356 | 0.505 | 0.803 | 0.320 | 0.435
VGG16/UNet | 0.780 | 0.565 | 0.646 | 0.616 | 0.557 | 0.570
ResNet34/UNet | 0.733 | 0.533 | 0.603 | 0.498 | 0.564 | 0.477

### Visual Results

Figure 5 shows sample visual results from different models. In total, 8 images
are presented, including two images for each pavement type on each viewpoint.
Black and Red colours are used to indicate true positive pixels and false-
positive pixels respectively.
The DeepCrack model generally captured all crack pixels in images regardless
of viewpoints and pavement types, although it also commonly misrecognized
concrete joints as cracks in all images. Visually, the VGG-16/UNet model has
the closest match to the ground truth segmentation mask. The ResNet-34/UNet
model demonstrated a very similar performance to the VGG-16/UNet model. The
Localized thresholding model has reasonable performance on some images, but
overall it failed to do the crack detection job on the NHA12D dataset.

### Discussion

Considering that the purpose of using crack detection algorithms is to
eliminate or sufficiently reduce the need to examine the raw images, it is
important to capture most cracks and avoid misclassifying healthy surfaces.
From this perspective, the three latest crack detection models perform poorly
on the proposed pavement dataset. All models were unable to differentiate
between cracks and concrete joints, leading to a high false-positive rate.
Also, some model failed to recognize obvious cracks on asphalt pavements,
making them unusable in real practice. The training dataset, which is a large
collection of all available crack datasets, contains very few images for
concrete pavements where joints and cracking co-exist. Consequently, such
knowledge cannot transfer to model. This highlights the need to collect and
label concrete pavement images for crack detection.
Another reason for the unsatisfactory performance is the difference between
the training datasets and testing datasets. The machine learning model
normally assumes that the training and testing sets come from the same
distributions/domain. Directly applying the trained model on the new dataset
may cause degradation in the performance. In our case, models were trained on
public crack datasets but then tested on new images in NHA12D dataset, which
is a typical Domain Shift challenge. To overcome these challenges, using
domain adaptation techniques such as generating synthetic data can boost
performance. Research in this area is less studied in current literature.

## Conclusions

Crack detection from 2D images has been investigated intensively and a large
number of papers have been published in the last two decades. However, fully
automated crack detection remains challenging due to the complex pavement
images captured in practice. To thoroughly examine the SOTA crack detection
algorithms, we proposed a new dataset that contains a more complicated
background and various types of pavement. Also, we conducted a robust
benchmark study of three the state of the art crack detection algorithms,
including DeepCrack, VGG-16/UNet and Resnet34/UNet, on the proposed dataset,
to quantitatively and objectively evaluate their performance. Experiments show
that there is still a gap in concrete pavement crack detection because current
models fail to separate concrete joints and cracks. From the outcome of the
benchmark study, we identify the potential directions in the future including,
a) producing the more well-labelled crack images on concrete pavements and
developing algorithms for concrete pavement images, b) applying domain
adaptation techniques to enhance the model performance on unseen images.

## References

##

  * (1)
  * Ai et al. (2018) Ai, Dihao, Guiyuan Jiang, Lam Siew Kei and Chengwu Li. 2018.  “Automatic Pixel-Level Pavement Crack Detection Using Information of Multi-Scale Neighborhoods.” IEEE Access 6:24452–24463.
  * Amhaz et al. (2016) Amhaz, Rabih, Sylvie Chambon, Jerome Idier and Vincent Baltazart. 2016\.  “Automatic Crack Detection on Two-Dimensional Pavement Images: An Algorithm Based on Minimal Path Selection.” IEEE Transactions on Intelligent Transportation Systems 17(10):2718–2729.
  * Attoh-Okine and Ayenu-Prah (2008) Attoh-Okine, Nii and Albert Ayenu-Prah. 2008.  “Evaluating pavement cracks with bidimensional empirical mode decomposition.” Eurasip Journal on Advances in Signal Processing 2008\.
  * Chen and Jahanshahi (2018) Chen, Fu Chen and Mohammad R. Jahanshahi. 2018.  “NB-CNN: Deep Learning-Based Crack Detection Using Convolutional Neural Network and Naïve Bayes Data Fusion.” IEEE Transactions on Industrial Electronics 65(5):4392–4400.
  * Dorafshan, Thomas and Maguire (2018) Dorafshan, Sattar, Robert J. Thomas and Marc Maguire. 2018.  “Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete.” Construction and Building Materials 186:1031–1045.
  * Dung and Anh (2019) Dung, Cao Vu and Le Duc Anh. 2019.  “Autonomous concrete crack detection using deep fully convolutional neural network.” Automation in Construction 99(November 2018):52–58.
  * Fan et al. (2018) Fan, Zhun, Yuming Wu, Jiewei Lu and Wenji Li. 2018.  Automatic Pavement Crack Detection Based on Structured Prediction with the Convolutional Neural Network. Technical report.
  * Hsieh and Tsai (2020) Hsieh, Yung-An and Yichang James Tsai. 2020.  “Machine Learning for Crack Detection: Review and Model Performance Comparison.” Journal of Computing in Civil Engineering 34(5):04020038.
  * Hu and Zhao (2010) Hu, Yong and Chun-xia Zhao. 2010.  “A Novel LBP Based Methods for Pavement Crack Detection.” Journal of Pattern Recognition Research 5(1):140–147.
  * Jenkins et al. (2018) Jenkins, Mark David, Thomas Arthur Carr, Maria Insa Iglesias, Tom Buggy and Gordon Morison. 2018.  A deep convolutional neural network for semantic pixel-wise segmentation of road and pavement surface cracks. In European Signal Processing Conference.  Vol. 2018-Septe European Signal Processing Conference, EUSIPCO pp. 2120–2124.
  * Katakam (2009) Katakam, Nikhil. 2009.  “Pavement Crack Detection System Through Localized Thresholding.” (August):2200.
  * Lau et al. (2020) Lau, Stephen L.H., Edwin K.P. Chong, Xu Yang and Xin Wang. 2020.  “Automated Pavement Crack Segmentation Using U-Net-Based Convolutional Neural Network.” IEEE Access 8:114892–114899.
  * Li et al. (2019) Li, Haifeng, Dezhen Song, Yu Liu and Binbin Li. 2019.  “Automatic Pavement Crack Detection by Multi-Scale Image Fusion.” IEEE Transactions on Intelligent Transportation Systems 20(6):2025–2036.
  * Li et al. (2016) Li, Peng, Chao Wang, Shuangmiao Li and Baocai Feng. 2016.  “Research on crack detection method of airport runway based on twice-threshold segmentation.” Proceedings - 5th International Conference on Instrumentation and Measurement, Computer, Communication, and Control, IMCCC 2015 pp. 1716–1720.
  * Liu et al. (2019) Liu, Wenjun, Yuchun Huang, Ying Li and Qi Chen. 2019.  “FPCNet: Fast Pavement Crack Detection Network Based on Encoder-Decoder Architecture.”.
  * Oliveira and Correia (2009) Oliveira, Henrique and Paulo Lobato Correia. 2009.  “Automatic road crack segmentation using entropy and image dynamic thresholding.” European Signal Processing Conference pp. 622–626.
  * Ragnoli, De Blasiis and Di Benedetto (2018) Ragnoli, Antonella, Maria Rosaria De Blasiis and Alessandro Di Benedetto. 2018.  “Pavement Distress Detection Methods: A Review.” Infrastructures 3(4).
  * S. Chambon (2011) S. Chambon, AigleRN. 2011.  “Results with AigleRN images.” https://www.irit.fr/~Sylvie.Chambon/AigleRN.html.  Accessed: 2021-07-10.
  * Shi et al. (2016) Shi, Yong, Limeng Cui, Zhiquan Qi, Fan Meng and Zhensong Chen. 2016.  “Automatic Road Crack Detection Using Random Structured Forests.” IEEE Transactions on Intelligent Transportation Systems 17(12):1–12.
  * Tang and Gu (2013) Tang, Jinshan and Yanliang Gu. 2013.  “Automatic crack detection and segmetnation using a hybrid algorithm for road distress analysis.” Proceedings - 2013 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2013 pp. 3026–3030.
  * Tsai, Kaul and Mersereau (2010) Tsai, Yi Chang, Vivek Kaul and Russell M. Mersereau. 2010.  “Critical assessment of pavement distress segmentation methods.” Journal of Transportation Engineering 136(1):11–19.
  * Wang, Li and Gong (2007) Wang, Kelvin C.P., Qiang Li and Weiguo Gong. 2007.  “Wavelet-based pavement distress image edge detection with a trous algorithm.” Transportation Research Record (2024):73–81.
  * Wang and Tang (2011) Wang, Shengchun and Wensheng Tang. 2011.  “Pavement crack segmentation algorithm based on local optimal threshold of cracks density distribution.”.
  * Yan et al. (2007) Yan, Maode, Shaobo Bo, Kun Xu and Yuyao He. 2007.  “Pavement crack detection and analysis for high-grade highway.” 2007 8th International Conference on Electronic Measurement and Instruments, ICEMI pp. 4548–4552.
  * Yang et al. (2020) Yang, Fan, Lei Zhang, Sijia Yu, Danil Prokhorov, Xue Mei and Haibin Ling. 2020.  “Feature Pyramid and Hierarchical Boosting Network for Pavement Crack Detection.” IEEE Transactions on Intelligent Transportation Systems 21(4):1525–1535.
  * Zhao, Qin and Wang (2010) Zhao, Huili, Guofeng Qin and Xingjian Wang. 2010.  “Improvement of canny algorithm based on pavement edge detection.” Proceedings - 2010 3rd International Congress on Image and Signal Processing, CISP 2010 2:964–967.
  * Zou et al. (2012) Zou, Qin, Yu Cao, Qingquan Li, Qingzhou Mao and Song Wang. 2012.  “CrackTree: Automatic crack detection from pavement images.” Pattern Recognition Letters 33(3):227–238.
  * Zou et al. (2019) Zou, Qin, Zheng Zhang, Qingquan Li, Xianbiao Qi, Qian Wang and Song Wang. 2019.  “DeepCrack: Learning hierarchical convolutional features for crack detection.” IEEE Transactions on Image Processing 28(3):1498–1512.