# Transfer Learning-based Road Damage Detection for Multiple Countries

Deeksha<EMAIL_ADDRESS>Centre for Transportation Systems
(CTRANS), Indian Institute of Technology Roorkee, 247667, India Institute of
Industrial Science, University of Tokyo, 4-6-1 Komaba, Tokyo, Japan  Hiroya
Maeda  Institute of Industrial Science, University of Tokyo, 4-6-1 Komaba,
Tokyo, Japan  Sanjay Kumar Ghosh  Centre for Transportation Systems (CTRANS),
Indian Institute of Technology Roorkee, 247667, India Department of Civil
Engineering, Indian Institute of Technology Roorkee, 247667, India  Durga
Toshniwal  Centre for Transportation Systems (CTRANS), Indian Institute of
Technology Roorkee, 247667, India Department of Computer Science and
Engineering, Indian Institute of Technology Roorkee, 247667, India  Alexander
Mraz  Institute of Industrial Science, University of Tokyo, 4-6-1 Komaba,
Tokyo, Japan  Amazon EU, 22 Rue Edward Steichen, 2540, Luxembourg  Takehiro
Kashiyama  Institute of Industrial Science, University of Tokyo, 4-6-1 Komaba,
Tokyo, Japan  Yoshihide<EMAIL_ADDRESS>Institute of
Industrial Science, University of Tokyo, 4-6-1 Komaba, Tokyo, Japan

###### Abstract

Many municipalities and road authorities seek to implement automated
evaluation of road damage. However, they often lack technology, know-how, and
funds to afford state-of-the-art equipment for data collection and analysis of
road damages. Although some countries, like Japan, have developed less
expensive and readily available Smartphone-based methods for automatic road
condition monitoring, other countries still struggle to find efficient
solutions. This work makes the following contributions in this context.
Firstly, it assesses usability of the Japanese model for other countries.
Secondly, it proposes a large-scale heterogeneous road damage dataset
comprising 26620 images collected from multiple countries using smartphones.
Thirdly, we propose generalized models capable of detecting and classifying
road damages in more than one country. Lastly, we provide recommendations for
readers, local agencies, and municipalities of other countries when one other
country publishes its data and model for automatic road damage detection and
classification. Our dataset is available at
(https://github.com/sekilab/RoadDamageDetector/).

22footnotetext: Abbreviations: API, Application Programming Interface, CNN,
Convolutional Neural Network, RDD, Road Damage Dataset, SSD, Single Shot
Detector

##  Introduction

Road infrastructure is a crucial public asset as it contributes to economic
development and growth while bringing critical social benefits. It connects
communities and businesses and provides access to education, employment,
social, and health services. However, road surface wears and deteriorates over
time from factors related to location, age, traffic volume, weather,
engineering solutions, and materials being used to build it and, therefore,
the knowledge of its deterioration extent are critical to efficient and cost-
effective maintenance with the goal to preserve its good and safe condition.
Many studies and surveys were made on the topic of roadway deficiencies and
their impact on safety and economy (Miller and Zaloshnja,  (2009) and
Radopoulou and Brilakis,  (2015)). Pavement distress, one of the pavement
condition characteristics, is typically evaluated using one of three
approaches: manual, semi-automated, or fully automated. The traditional
methods for obtaining pavement condition data include manual and semi-
automated surveys. In manual surveys, raters perform a visual inspection of
the pavement surface through either walking on or along the pavement surface
or by conducting a windshield survey from a slow-moving vehicle. Visual
inspection of the road surface suffers from a subjective judgment of
inspectors. It requires a significant human intervention that is proven to be
time-consuming, given the extensive length of road networks. Moreover,
inspectors must often be physically present in the travel lane, exposing
themselves to potentially hazardous conditions. In semi-automated pavement
condition evaluations, the road images are collected automatically from a
fast-moving vehicle, but distress identification is postponed to an off-line
process running in workstations at the office. This approach improves safety
but still uses manual distress identification, which is very time-consuming.
Currently, most state highway agencies use the semi-automated method (Zalama
et al.,  (2014); Pierce et al.,  (2013); McGhee,  (2004)), which involves some
degree of human intervention. Fully automated distress evaluations often
employ vehicles equipped with sophisticated and expensive sensors. The
processing of collected data is then conducted using image processing and
pattern recognition software for distress identification and quantification.
The data processing may be accomplished during data collection or in post-
processing at the office. Raters’ role is to conduct quality assurance testing
of the software functionality and perform quality control of the distress
rating output by the software. Unlike the advanced data collection technology
available to the transportation agencies, the pattern recognition software
still needs further enhancements to detect and classify the various types of
pavement surface distresses on different pavement surfaces to the accuracy
levels acceptable to the agencies. Only a few highway agencies have
implemented a fully automated crack detection system for network-level data
collection. From experience, a systematic quality management process seems to
be the central consideration for successful implementation (Kargah-Ostadi et
al.,  (2017)). Specialized vehicles used for pavement inspection are usually
equipped with multiple sensors such as laser scanners, road profilers, and
cameras with the aim to capture road assets, including pavement images, and to
acquire the longitudinal and transverse profiles of the road (Roadware,
(2019)). However, such vehicles are expensive, and the cost to purchase such
systems can reach half a million dollars, depending on the sensors included.
Even though the operating price of road survey vehicles is cheaper than that
of traditional surveying methods, it can reach between $30 and $50 per
kilometer (Radopoulou and Brilakis,  (2015)). Meanwhile, mobile devices such
as smartphones have evolved in recent years into devices containing high-
resolution digital cameras, sensors, and powerful processors; therefore,
examples of road inspection using smartphones are becoming more common. The
advantage of employing smartphones is that they enable efficient and cost-
effective inspection of the road surface of large road networks. For example,
Casas-Avellaneda and López-Parra,  (2016) proposed a method to visualize
pothole detected by smartphone sensors on a map. Mertz et al.,  (2014)
proposed a method to handle road images acquired by on-board smartphones
installed on cars that operate daily, such as general passenger automobiles,
buses, and garbage trucks, to detect road surface damage with an external
laptop. Maeda et al.,  (2018) developed a smartphone application for the
collection and real-time detection of roadway deficiencies used by Japanese
local governments. Even though Japanese Municipalities have started using the
Smartphone-based model, there are many municipalities and local authorities in
several other countries still lacking the technology, know-how, and funds to
afford expensive state-of-the-art data collection equipment. On many
occasions, they sustain their road networks through inefficient maintenance
planning, and in some circumstances, they are unable to carry out adequate
inspections. In the context of this challenging situation regarding the
maintenance of such infrastructure, there is a need to assess the usability of
the Japanese Road Damage Detection and Classification model for monitoring the
road conditions in other countries. Further, it needs to be analyzed whether
there would be any effect on the accuracy level of the Japanese model when
used for other countries. If yes, what are the options available for other
countries if the performance of the Japanese model is not found satisfactory
when applied to their local roads? Do they need to train a new model for
themselves from scratch, or can they utilize the Japanese model or data in
some other way? The above-mentioned questions need to be addressed based on
some detailed research. Further, the recommendations are often sought for
readers of other countries when one country, in this case, Japan, publishes
its data or model for a specific application. The presented work intends to
answer the aforementioned research questions and provide the readers with apt
recommendations regarding the automation of road damage detection and
classification. For the required comprehensive analysis, this study considers
two countries, namely India and the Czech Republic (partially Slovakia), and
conducts several experiments using different combinations of underlying data.

First, localized road damage datasets were created by using 3590 road images
from Czech and 9892 images from India using smartphones installed on the
windshield of the vehicle. The newly collected datasets were labeled for
cracks and potholes present in the images and were mixed with the Japanese
dataset. Next, the study involves training and evaluating 16 deep neural
network models considering 30 scenarios based on different combinations of the
test and train datasets for detecting and classifying road damages. The
evaluation results of these models are analysed to provide recommendations for
the readers of other countries.

Overall, the contributions of this work can be listed as follows:

  1. 1.

Performance analysis of the usability of the Japanese model for detecting and
classifying the road damages in other countries.

  2. 2.

Proposing a large-scale heterogeneous dataset, comprising $26620$ annotated
road images collected using a vehicle-mounted Smartphone from multiple
countries.

  3. 3.

Proposing generalized hybrid models capable of detecting and classifying road
damages in more than one country.

  4. 4.

Providing recommendations for readers, local agencies/municipalities of other
countries when one country publishes its data and model for automatic road
damage detection and classification.

The following section provides an overview of the existing literature.

##  Related Work

###  Machine Learning and Convolutional Neural Networks

Figure 1: Typical CNN Architecture

The research area of neural networks is a successful field within computer
science, which specializes in providing solutions in application domains that
are difficult to model with conventional statistical approaches(Adeli,
(2001)). Such applications are usually characterized by noisy input data,
largely unknown intrinsic structure, and changing conditions (Hasenauer et
al.,  (2001)). The significant difference between neural networks and
traditional methods of computer science is that the behavior of the neural
networks is the result of a training process. In contrast, in traditional
methods, the behavior of the system is predefined. Neural networks consist of
artificial neurons that have learn-able weights and biases. Convolutional
Neural Network (CNN) is one of the main categories of deep neural networks
that are usually applied for image recognition and classification. These are
the type of algorithms that can identify street signs, cars, faces, and many
other types of objects. Each artificial neuron in CNN is activated after
processing the input image using various convolutional operations, such as
gradient, edge detection filters, blobs, in combination with learn-able
weights and biases (Peppa et al.,  (2018)). An example of a typical CNN
architecture consists of series of convolution and pooling layers designed for
feature extraction and fully connected layers and the activation function for
classification of objects with probabilistic values ranging from 0 to 1, as
shown in 1. The main advantage of CNN is that after the training stage, it
automatically detects the critical features without any human supervision.

The core building block of CNN is a convolution layer that employs convolution
operations that are applied to the input data using convolution filters, also
called kernels, to produce an output called feature map. Convolution of an
image with different filters is done for operations such as edge detection,
blur, or sharpening. It uses small squares of input data to learn the image
features and preserves the correlation between pixels (Prabhu,  (2018)). As a
result, the network determines which filters activate when the neural network
detects some specific type of feature at some spatial position within the
input image. Pooling layers are usually inserted between successive
convolution layers, and their function is to progressively reduce the number
of parameters in the network, which both shortens the training time and
combats overfitting. It is a form of down-sampling. Contrary to convolution
operation, pooling has no parameters, and it slides a window over its input
and takes, for example, the maximum value in the windows (Dertat,  (2017)). A
fully connected layer performs high-level reasoning in the neural network.
Neurons in the fully connected layer have a connection to all activations from
neurons in the previous layer, like regular artificial neural networks.
Activation function, also known as the transfer function, is used to map
resulting values of the neural network, which may be negative or greater than
one, into an interval between 0 and 1. For multi-class classification, a
softmax function, which is a more generalized logistic activation function, is
implemented (Sharma,  (2017)).

Figure 2: Comparison of (a) Regular and (b) Depth-wise Separable Convolutions
applied on 3-channel (RGB) Image

Many CNN architectures, such as R-CNN (Region-based Convolutional Neural
Networks;Girshick et al.,  (2014)), Fast-CNN(Girshick,  (2015)), or Faster-CNN
(Ren et al.,  (2015)) have been developed to attain the best accuracy while
improving the processing speed. However, the computational load was still too
large for processing images on devices with limited computation, power, and
space (He et al.,  (2016)). Therefore, the Single Shot Multi-Box Detector(SSD)
framework was developed to improve further the computation speed of the object
detection (Liu et al.,  (2016)). It uses a single feed-forward convolutional
network to detect multiple objects within the image directly and combines
predictions from numerous feature maps with different resolutions to handle
objects of various sizes. MobileNet is a small, low-latency, and low-power
convolutional feature extractor that can be built to perform classification,
detection, or segmentation similar to popular large-scale models, such as
Inception SSD (Szegedy et al.,  (2016)). It is based on depth-wise separable
convolution, which factorizes a standard convolution into a depth-wise
convolution, and $1\times 1$ convolution called a point-wise convolution.
Depth-wise convolution maps a single convolution on each input channel
separately, and point-wise convolution is a convolution with a kernel size of
$1\times 1$ that combines the features created by the depth-wise convolution
(Douillard,  (2018)). In comparison to depth-wise separable convolution, a
regular convolution does both filtering and combination steps in the single
run. However, SSD MobileNet requires more computational work to accomplish the
task, and it needs to learn more weights (Figure 2). More information on
MobileNet is given in a paper published by Howard et al.,  (2017). Huang et
al.,  (2017) compared many neural networks and object detection methods, and
among them, the SSD object detection framework using a MobileNet base network
was found to require relatively small CPU load and memory consumption while
maintaining high accuracy. Budzar,  (2018) compared the accuracy of the famous
large-scale Inception model to MobileNet and concluded that the difference was
less than 6% (95.2% vs. 89.5%, respectively) while MobileNet was more than
six-times smaller when compared to Inception model(13 MB vs. 87 MB). MobileNet
has also been shown to achieve accuracy comparable to VGG-16 on the ImageNet
dataset with only 1/30th of the computational cost and model size (Maeda et
al.,  (2018)). MobileNet is designed to effectively maximize accuracy while
being mindful of the restricted resources for on-device or embedded
applications. Due to all these advantages, the study presented in this paper
adopted SSD MobileNet.

###  Road Infrastructure Inspection using Machine Learning

Recent studies have adopted various deep learning neural network systems for
automated road surface survey or damage detection(Cha et al.,  (2018), Zhang
et al.,  (2017), Lin et al.,  (2017)). For example, Silva and Lucena,  (2018)
developed a machine learning-based model to detect cracks on concrete
surfaces. Supervised deep convolutional neural network was trained by Zhang et
al.,  (2016) to classify pavement images taken by smartphones around the
Temple University campus. However, proposed road damage detection methods
focused only on the determination of the existence of damage. Anand et al.,
(2018) presented the development of deep neural network architecture using GPU
board and associated camera to detect road cracks and potholes for self-
driving cars and autonomous robots to perform necessary evading maneuvers to
ensure a smooth journey. Supervised, convolutional neural network (CNN) was
trained by Fan et al.,  (2018) to recognize different pavement conditions, and
results presented in the paper show that it can deal well with varying
pavement textures. The detection method developed by Maeda et al.,  (2018)
classified pavement deterioration of Japan road network into eight categories
based on images captured by mobile devices with a resolution of 300 x 300
pixels. The corresponding dataset, named RDD-2018, was made publicly available
in 2018, and a Smartphone-based application was also introduced for real-time
road condition assessment. Since then, this application is being used by
several municipalities in Japan for faster monitoring of road conditions. As a
result, the underlying data, method, and models have gained wide attention
from researchers all over the world. A technical challenge was organized in
December 2018 as a part of the IEEE Big Data Conference held at Seattle, USA,
which utilized this data for evaluating the performance of several models for
road condition monitoring. In total, 59 teams participated in the challenge
from 14 different countries. These teams, although provided solutions having
better accuracy than the original models included in Maeda et al.,  (2018),
the new solutions were mostly based on changing the underlying network models
and hyper-parameter configurations (Alfarrarjeh et al.,  (2018); Kluger et
al.,  (2018); Wang et al., 2018b ; Wang et al., 2018a ). Although some authors
suggested improvements in the dataset but the teams themselves did not modify
the dataset in their work. After that, some researchers focused on
experimenting with the underlying dataset by either adding more images to it
or by introducing a completely new dataset based on a similar method utilized
by Maeda et al.,  (2018). For instance, Angulo et al.,  (2019) extended this
dataset by adding images collected from Italy and Mexico. The size of the
dataset was increased to 18034 images, and the authors mostly focused on
having a more balanced representation for individual damage classes,
especially potholes. Similarly, Roberts et al.,  (2020) used the data
collection app introduced by Maeda et al.,  (2018) and collected over 7000
road images from Sicily, Italy. Their work utilizes only the newly collected
images, but the underlying methodology is similar to Maeda et al.,  (2018),
except that they also consider severity analysis for the identified road
damages. The presented study differs from these works in the following ways:

  1. 1.

The Smartphone-based dataset introduced in our work is larger and more
heterogeneous, covering multiple countries.

  2. 2.

These works do not consider the perspective of evaluating the responses of
existing or newly proposed models for detecting damages in different countries
individually.

  3. 3.

The Japanese dataset has been updated by re-annotating and adding more images
from Japan, named RDD-2019, in recent work (Maeda et al.,  (2020)). Unlike the
previous studies based on the Japanese dataset (2018 version) or models, our
work builds upon this updated dataset.

  4. 4.

We also analyze the effect of varying the number of images in training dataset
on the performance of the trained models. This also lays the foundation for
analyzing whether a model trained on data from a single country or multiple
countries performs better.

Another work (Du et al.,  (2020)) uses a dataset of 45,788 road images
collected from Shanghai and utilize YOLO network for detecting and classifying
pavement distresses. However, the images are collected using an industrial
high-resolution camera, unlike our work, which is based on the lesser
expensive Smartphone-based images. Similarly, Majidifard et al.,  (2020) use
Google street view images considering both top-down as well as wide-view for
classification and densification of pavement distresses collected from 22
different pavement sections in the United States. However, the size of the
dataset used is limited to only 7237 images. Ideally, more than 5000 labeled
images are generally required for each class for an image processing based
classification task to provide satisfactorily accurate results (Goodfellow et
al.,  (2016); Maeda et al.,  (2020)). This is one of the reasons why there is
a need to analyze whether the data already available from another country can
be mixed with the local data for improving the performance of a road damage
detection and classification model. This study intends to address the research
gaps and provides the recommendations required for practical use of the models
by different countries. The following section provides detail of the proposed
dataset and the methodology followed by this study to design the experiments
and carry out the requisite analysis.

##  Datasets and Methodology

This section first explains the transition from the dataset RDD-2018 to the
proposed dataset, followed by an explanation of the dataset used in this
study.

###  RDD-2018

Table 1: Road Damage Types and Definitions considered in Maeda et al.,  (2018)

Damage Type | Detail | Class Name
---|---|---
|  | Longitudinal | Wheel-marked part | D00
Crack | Linear Crack |  | Construction joint part | D01
|  | Lateral | Equal interval | D10
|  |  | Construction joint part | D11
| Alligator Crack | Partial pavement, overall pavement | D20
|  |  | Pothole | D40
Other Damage | Cross walk blur | D43
|  |  | White line blur | D44

RDD-2018 (Maeda et al.,  (2018)) was introduced and made publicly available in
2018, and since then, it has been used exhaustively by several researchers to
propose new methods for automatic condition monitoring of roads as well as
other infrastructures. Table 1 shows the damage categories and the
corresponding definitions used in this dataset.

###  RDD-2019

Maeda et al.,  (2020) reviewed and relabelled the contents of the dataset
RDD-2018 and added new annotated images to form the new dataset RDD-2019. The
number of images with annotations were increased from 9,053 to 13,135. The
number of annotations increased from 15,435 to 30,989.

###  The proposed Road Damage Dataset 2020

The proposed dataset builds upon the recently introduced RDD-2019 dataset.
This is to be noted that the road damage data included in Road Damage Dataset
2018 and 2019 comprise road images collected from only a single country, that
is, Japan. However, the dataset proposed in this study considers data
collected from multiple countries. The significant points of difference are as
follows.

  1. 1.

The total number of images is increased to 26620, almost thrice the prevailing
2018 dataset.

  2. 2.

New images were collected from India and Czech Republic (partially from
Slovakia) to make the data more heterogeneous and train robust algorithms.

  3. 3.

Unlike previous versions, this dataset considers only four damage categories,
comprising mainly of cracks and potholes, namely D00, D10, D20, and D40. Note
that the standards related to evaluations of Road Marking deterioration such
as Crosswalk or White Line Blur differ significantly across several countries.
Thus, these categories were excluded from the study so that generalized models
can be trained applicable for monitoring road conditions in more than one
country.

More details related to the newly collected images from India and the Czech
Republic are provided as follows.

####  3.3.1 Study Area

Road images for the presented study were collected from India, the Czech
Republic and partially, in Slovakia. For the Czech Republic and Slovakia, a
large portion of road images was collected in Olomouc, Prague, and Bratislava
municipalities and included a mix of 1st class, 2nd class, 3rd class roads,
and local roads. A smaller portion of the road image dataset was collected
along D1, D2, and D46 motorways to improve the robustness of the trained
model. Similarly for India, a mixture of images collected from local roads,
State Highways, and National Highways was considered, covering the
Metropolitan (Delhi, Gurugram) as well as Non-Metropolitan regions (mainly
from the state Haryana). All these images were collected from plain regions.
Road selection and time of data collection were made based on road
accessibility, weather conditions, and traffic volume.

####  3.3.2 Data Collection

Road images were captured using a smartphone running a publicly available
image-capturing application developed by Sekimoto Lab, The University of Tokyo
(Maeda et al.,  (2018)). The smartphone was installed on the windshield inside
the vehicle, and the application captured JPEG images with a resolution of
$600\times 600$ pixels at a rate of one image per second (Figure 3). For
India, the updated version of the application was used, and the images with a
resolution of $720\times 960$ pixels were captured. These images were resized
to $720\times 720$ to have square images similar to Japan and Czech to
maintain uniformity in the dataset for different countries. This rate was
selected to enable the collection of pictures without overlap or leakage when
the vehicle traveled at an average speed of approximately 40 km/h (or 25 mph).
However, vehicle speed on motorways and some 1st class roads/highways outside
the cities was adjusted to comply with the local laws.

For the Czech Republic and Slovakia, a total of 10400 images were collected
between March and October 2018 under varying weather and lighting conditions,
including sunny, overcast, light rain, and sunset. However, only 3595 images
were found suitable for the study. The remaining images which were either
blurred or were not covering the required road damage categories or
significant portions of the road were discarded. Similarly, for India, a total
of 10595 images were captured in October 2019, out of which, 9892 images were
finally selected.

Figure 3: Installation Setup of the Smartphone in the Car

####  3.3.3 Data Classification/Damage Categories

Based on the Japanese Maintenance Guidebook for Road Pavement (JRA, 2013),
Maeda et al.,  (2018) use eight damage categories in total. This
classification covers two types of deterioration, namely pavement
deterioration (D00, D01, D10, D11, D20, D40) and road marking deterioration
(D43, D44), as shown in Table 1. However, the standards for evaluating the
road pavement conditions are not the same across the countries and vary
significantly. For instance, the Czech Catalog of Deficiencies in Flexible
Pavements, Czech Ministry of Transport(2009) recognizes 28 different pavement
deterioration divided into three main categories, including loss of asphalt
mixture, cracks, and deformations. This catalog does not recognize Crosswalk
marking deterioration as a type of pavement deficiency. Similarly, for India,
the code IRC:82,  (2015) provides guidelines for the maintenance of
bituminous/flexible pavement in India. It identifies four main types of
pavement distresses, namely, Cracking, Deformation (rutting, upheavals, etc.),
Disintegration (raveling, potholes, edge breaking, etc.), and Surface Defects
that relate to inapt quality and quantity of bitumen. Since the aim of this
study is to consider the perspective of monitoring road conditions in more
than one country, we include only the four major categories of damage mainly
comprising cracks and potholes, which are generally common to almost all the
countries. Thus, for this study, the damage categories considered are - D00 to
represent Longitudinal/ Parallel cracks, D10 for Transverse/Perpendicular
cracks, D20 for Alligator/Complex cracks, and D40 for Potholes. Some other
categories which are indistinguishable in the images from the above mentioned
four cases, for instance, rutting is marked as D40 in Japanese dataset (Maeda
et al.,  (2018)), are covered along with these so as to provide a generalized
model. Further, this is to be noted that the existing Japanese datasets
(RDD-2018 and 2019) also contain annotations for damage categories (D01, D11,
D43, D44, D50), which are not included in this study. Our work makes no
changes to these already available annotated images. Instead, it uses
algorithms that read only the annotations related to the desired four
categories.

###  Data Annotation

Figure 4: Annotation Pipeline: (a) original image, (b) image with bounding
boxes, (c) final annotated image containing bounding boxes and class labels

Data annotation is a fundamental part of what makes many machine learning
projects function accurately. It provides the initial setup for teaching a
deep neural network to recognize objects and discriminate them against various
input images. In this work, the collected road images were manually annotated
using labelImg software. The annotation pipeline is presented in Figure 4.
Annotation is a labor-intensive procedure and took considerable time because
the complete dataset was first manually reviewed on the screen and then all
recognized deficiencies were annotated by enclosing them with bounding boxes
and classified by attaching the proper class label. Class labels and bounding
box coordinates, defined by four decimal numbers
$(x_{min},y_{min},x_{max},y_{max})$, were stored in the XML format similar to
PASCAL VOC (Everingham et al.,  (2010)). Next, the data was converted into
TFRecord file format, as required by the TensorFlow Object Detection API
(Intel,  (2018)).

####  3.4.1 Statistics

Figure 5: Statistics for the number of damage instances included in the
underlying datasets

Figure 5 shows the statistics for the proposed dataset. A comparatively lesser
number of images are used from the countries India and Czech than what was
available from Japan in RDD-2019. This is because our motive is to analyze and
provide solutions in which the dataset already available from other countries
can be utilized to design models for new countries, such that the requirement
for collecting new local data is minimized. Further, this can be noted that
the number of instances for different damage categories in Indian and Czech
datasets is a bit dis-balanced. This leads to the requirement of analyzing if
the data from another country can be used to create a balanced representation.
For example, only 89 instances of transverse cracks, that is, D10 are present
in the 9892 images available from India, which is very less. Combining it with
dataset available from Japan may help in representing this class better while
training the requisite neural network models. Similarly, for alligator cracks,
D20, and potholes, D40, in the case of Czech data, adding data from another
country or collecting new local data becomes a necessity for training
efficient models. The following section illustrates the methodology followed
in this study.

###  Experimental Setup and the Methodology

Table 2: Naming Convention for the Models

Model Name | Training Data (#images) | Tested for
---|---|---
Japan | India | Czech
Ja_2k | 2000 |  |  | Japan, India, Czech
Ja_4k | 4000 |  |
Ja_6k | 6000 |  |
Ja_8k | 8000 |  |
Ja_10k | 10000 |  |
In_2k |  | 2000 |  | India
In_4k |  | 4000 |
In_6k |  | 6000 |
In_8k |  | 8000 |
Cz_2k |  |  | 2000 | Czech
InJa_12k | 10000 | 2000 |  | India & Japan
InJa_14k | 10000 | 4000 |
InJa_16k | 10000 | 6000 |
InJa_18k | 10000 | 8000 |
JaCz_11k | 10000 |  | 1000 | Czech
JaCz_12k | 10000 |  | 2000

The study involves training and evaluating 16 deep neural network models
considering 30 scenarios based on different combinations of the test and train
datasets, as described in table 2. The critical differences in the models are:

  1. 1.

Source of the training data: Following two cases were considered -

    1. (a)

Single Source Models – It covers the models trained on a single source of data
(Japan) applied to test data from different countries (India, Japan, Czech).

    2. (b)

Multiple Source Models – It covers the following two categories:

      1. i.

Pure Modelling or Same Source, Same Target Modelling: A model trained and
tested on data collected from the same country.

      2. ii.

Mixed Modelling: It includes mixing the local data of some other country,
generally the target country, with publicly available data from Japan for
training the models.

  2. 2.

Size of the training data: The number of images considered for training the
models was varied from 2000 to 18000 based on the availability of the data
from different countries. This was done to analyze the effect of varying the
size of training data on the performance of the models.

Further, since the images available from different sources were originally in
different resolutions as described in data collection, images were re-scaled
to a resolution of $300\times 300$ pixels while training the models.
TensorFlow API $1.80$ and official Python distribution 3.6.5 were used to
construct, train, and deploy the object detection models based on transfer
learning using MobileNet. The learning rate for training the models was set to
$0.003$.

##  Results and Analysis

###  Evaluation Parameters

The performance of the trained model was estimated by using three indices,
namely, precision, recall, and F1-score. Precision indicates the percentage of
correctly predicted features (i.e., true positives) out of the total number of
predicted features (i.e., true and false positives). The recall shows the
percentage of correctly predicted features out of the total number of features
present in the actual class (i.e., true positives and false negatives).

Figure 6: An illustration for calculating Intersection over Union (IoU)

Precision and recall are both based on evaluating Intersection over Union
(IoU), which is defined as the division of the area overlap between predicted
and ground-truth bounding boxes by the area of their union, as shown in Figure
6 . Because precision and recall counter each other and increasing one of them
will usually reduce the other, The most common way to reach a balance between
these metrics is to use F1-score. It measures the overall model’s accuracy and
is calculated as:

$F_{1}=2\times{\frac{(precision\times recall)}{(precision+recall)}}$ (1)

Maximizing the F1-score ensures reasonably high precision and recall.

For evaluation of the model’s performance, the IoU threshold was set to $0.5$,
which represents an evaluation metric prescribed by the PASCAL VOC object
detection competition (Everingham et al.,  (2010)).

###  Evaluation results

Table 3: F1-Score for the experiments

|  |  | Damage Category
---|---|---|---
Experiment | Model Name | Test Data(1000 images) | D00 | D10 | D20 | D40
E-1 | Ja_2k |  | 0.196 | 0.084 | 0.404 | 0.305
E-2 | Ja_4k |  | 0.257 | 0.138 | 0.450 | 0.305
E-3 | Ja_6k |  | 0.268 | 0.154 | 0.421 | 0.356
E-4 | Ja_8k |  | 0.293 | 0.194 | 0.435 | 0.334
E-5 | Ja_10k | Japan | 0.308 | 0.228 | 0.504 | 0.401
E-6 | Ja_2k |  | 0.024 | nan | 0.050 | 0.015
E-7 | Ja_4k |  | nan | nan | 0.022 | 0.048
E-8 | Ja_6k |  | 0.030 | nan | 0.034 | 0.010
E-9 | Ja_8k |  | 0.032 | nan | 0.022 | 0.052
E-10 | Ja_10k | India | 0.051 | nan | 0.028 | 0.036
E-11 | Ja_2k |  | 0.110 | 0.062 | 0.064 | Nan
E-12 | Ja_4k |  | 0.156 | 0.082 | 0.019 | Nan
E-13 | Ja_6k |  | 0.158 | 0.022 | 0.046 | Nan
E-14 | Ja_8k |  | 0.120 | 0.109 | 0.045 | Nan
E-15 | Ja_10k | Czech | 0.175 | 0.097 | 0.022 | Nan
E-16 | In_2k |  | 0.141 | nan | 0.432 | 0.266
E-17 | In_4k |  | 0.140 | nan | 0.448 | 0.295
E-18 | In_6k |  | 0.199 | nan | 0.502 | 0.333
E-19 | In_8k | India | 0.226 | nan | 0.442 | 0.307
E-20 | Cz_2k | Czech | 0.263 | 0.071 | 0.282 | 0.196
E-21 | InJa_12k |  | 0.159 | nan | 0.325 | 0.276
E-22 | InJa_14k |  | 0.129 | nan | 0.391 | 0.257
E-23 | InJa_16k |  | 0.200 | nan | 0.403 | 0.311
E-24 | InJa_18k | India | 0.224 | nan | 0.487 | 0.305
E-25 | InJa_12k |  | 0.327 | 0.174 | 0.522 | 0.369
E-26 | InJa_14k |  | 0.361 | 0.177 | 0.502 | 0.434
E-27 | InJa_16k |  | 0.343 | 0.111 | 0.512 | 0.366
E-28 | InJa_18k | Japan | 0.335 | 0.128 | 0.524 | 0.336
E-29 | JaCz_11k |  | 0.324 | 0.177 | 0.204 | 0.076
E-30 | JaCz_12k | Czech | 0.318 | 0.128 | 0.228 | 0.092

Table 3 presents the value of the F1-score obtained for different damage
categories in several experiments, respectively. The highlighted entries show
the maximum F1-score obtained for each country for different damage
categories.

###  Empirical Analysis

####  4.3.1 Single Source Modeling

Figure 7: Damage category-wise and Consolidated F1-Score for Models trained on
Japanese data applied to test data from different countries.

Figure 7 shows the results for different damage categories when a model
trained on Japanese data is applied to different target countries by using
test data constituting 1000 images from Japan, India, and Czech, respectively.

As depicted by the individual and the consolidated chart, the model trained on
Japanese data, though worked fine for Japanese test data; the performance is
abysmal when applied to test data from India and Czech, irrespective of the
number of images used for training. Though, the model performs better for some
damage categories viz. a viz. others, the overall performance is below
acceptable limits.

####  4.3.2 Multiple Source Modeling

The study analyzes the models trained using multiple sources of data based on
the target or the test data, as illustrated below.

##### Target: Japan

Figure 8: Model name vs F1-Score (Target: Japan)

Figure 8 shows the results for various models applied to Japanese test data.
Clearly, the models having training data as a mixture of images from India and
Japan show better performance for Japan for D00, D20, and D40. For D10, the
performance shows a downfall on increasing the number of images added from
India because the number of instances for the D10 category is almost
negligible in Indian data (89 in total). And, thus, as the total number of
images increases on adding Indian data to Japanese data, the proportion of D10
instances in the total number of damages decreases, leading to the decline in
the performance of the final trained model in identifying D10.

##### Target: India

Figure 9: Model name vs F1-Score (Target: India)

Figure 9 shows the compiled results for different models applied to test data
from India. The results illustrate that a model trained using Indian data
(either pure or mixed in the form of InJa models) perform significantly better
than the model trained using only Japanese data. Notably, even the models
trained using Indian data fail to perform satisfactorily for Transverse Cracks
(D10) in Indian data. This is because only 89 instances of transverse cracks
are present in the 9892 images available from India, leading to the presence
of only 9 to 10 images in Indian test data constituting 1000 images in total.
These 10 images are not sufficient for evaluating the performance of the
object detection models leading to nan(Not a number) values in F1-score. For
other categories, the observed trend show adding images from Japan does not
help much in improving the performance of the models trained using only Indian
data. Nonetheless, the performance of pure Indian models is slightly better
than the mixed InJa models, despite having smaller training datasets.

##### Target: Czech

Figure 10 shows the results for different models applied to the test data from
the Czech. Analyzing the graph from right to left shows, a model trained using
only 2000 images of Czech (Cz_2k) shows better average performance than the
model trained using Japanese data constituting up to 10000 images. However,
this may be a case of overfiting. Further, mixing the images from Czech with
the already available Japanese data (10000 images) shows some improvement over
the case when using data from a single country. Two cases were considered for
the analysis, based on the number of images used from Czech. The observations
are listed as follows:

  1. 1.

Adding images from Japan to Czech data helps in improving the performance for
Linear Cracks, both longitudinal as well as transverse.

  2. 2.

For Alligator cracks and Potholes – the pure Czech model seems to outperform
the mixed JaCz models. This can be explained as follows. The number of
instances for D20 and D40 in Czech data is very less, and mixing a huge amount
of Japanese data for training the model decreases the proportion of Czech D20
and D40 instances further. This leads to the dominated effect of corresponding
features learned from Japanese data and ultimately to degraded performance of
JaCz models for identifying D20 and D40 of Czech.

Figure 10: Model name vs F1-Score (Target: Czech)

###  Visual Analysis

Figure 11: Labels predicted for Sample Image 1 by models trained using
RDD_India and RDD_InJa

Figure 12: Labels predicted for Sample Image 2 by models trained using
RDD_India and RDD_InJa

As described in the empirical analysis, the behavior of models trained for
Japan or Czech is easily understandable. However, the models trained using
only the Indian datasets show comparable performance to the models trained
using the Indian dataset mixed with 10000 Japanese images, which is a bit
unusual. Usually, the performance should get improved on increasing the size
of the datasets by such a large extent. To better understand this uncommon
behavior of the models, we analyzed the corresponding prediction results
visually. Figure 11 and 12 show the labels predicted by several models trained
using Indian (In_2k, In_4k, In_6k, In_8k) and combined Indian-Japanese
(InJa_12k, InJa_14k, InJa_16k, InJa_18k) datasets for different Indian road
images.

The image in figure 11 shows a high severity alligator crack intermixed with
pothole-like structures. For the given image, India_8k identified that the
alligator cracks have taken the form of potholes. InJa_14k could identify the
presence of D40 with D20, though not individually. InJa_18k provided the most
precise result.

Clearly, there cannot be a single correct label boundary for this type of
image. The labels predicted by all the models are acceptable when a
comprehensive survey for monitoring road conditions is required. However,
since the evaluation of object detection models involves comparing the
predicted labels with the information in the ground truth file, some of the
predicted labels, although acceptable, get marked as false examples. When many
such images are present in the dataset, it results in low performance of some
of the models. This may be a reason for the low accuracy of InJa viz.a.viz.
Indian models, despite having larger training datasets. The features learned
by InJa models are a bit dominated by the Japanese dataset since the number of
images used from Japan is more than that of India in all the cases.

For the image in figure 12, the outputs of Indian models are too much
specific, providing a hint that these models may be overfitting the Indian
data. For InJa models, InJa_14k shows better accuracy than the remaining
models. To analyze if the Indian models are overfitting the Indian data, we
analyzed the behavior of these models for data collected from Japan and Czech.

Figure 13: Sample images from Japan and Czech for comparing the performances
of In_8k and InJa_18k models (Cases of missed detection by In_8k)

Figure 14: Sample images from Japan and Czech for comparing the performances
of In_8k and InJa_18k models (Cases of false detection)

Figure 13 shows some sample images from Japan and Czech in which In_8k missed
some damages, but InJa_18k identified them correctly. Figure 14 shows the
images for false detection by Indian or InJa models. From top to bottom, the
first image demonstrates the falsely detected D20 on the roadside by the InJa
model. However, the pattern is relatable to D20. For the second image, the
pothole-like structures were missed entirely by In_8k and mis-classified by
InJa_18k. Thus, in case the motive of the survey is to detect the presence or
absence of the damages, the performance of InJa_18k can be counted as better
than In_8k. The last image shows the case of false-positive detection by
India_8k.

In summary, it can be said that although the models trained using only Indian
data show performance comparable to models trained by mixing Japanese data
with Indian datasets (InJa), InJa models are more generalized. Also, for InJa
models, InJa_14k is found to have the best overall performance, combining the
evaluations for both India and Japan, and can be used efficiently for
detecting and classifying road damages in both the countries.

##  Discussion

Road maintenance plays an important role in the socioeconomic development of a
country. It requires regular assessment of road conditions, which is usually
carried out individually by several state agencies. Some agencies perform
pavement condition surveys using road survey vehicles equipped with a
multitude of sensors to evaluate pavement conditions and deterioration. In
these vehicles, laser line-scan cameras and three-dimensional($3D$) cameras
are generally employed to capture pavement surfaces with the best possible
quality and resolution. However, such imaging equipment mounted on dedicated
vehicles is expensive and is often un-affordable for local agencies with
limited budgets. This leads to the requirement of low-cost methods capable of
comprehensively surveying road surfaces, such as methods that can be
implemented using Smartphones. The municipalities in Japan have already
started using such Smartphone-based applications; however, many other
countries still lack proper research in this direction. This work is an
attempt to systematically analyze the opportunities available for these
countries when one other country has developed such an application and has
made the underlying data and models available publicly. The study involves
training 16 models evaluated using 30 experiments based on the proportion of
data used from different countries. Notably, the performance of the model
trained using single-source of data, in this case, Japan, varies for different
targets significantly, indicating that the same model cannot be used
everywhere. For multiple source models (pure vs. mixed modeling), the
performance varies based on the target country for which the model is trained.
For Japan, adding images from India to the Japanese dataset helps in improving
the performance for longitudinal cracks, alligator cracks as well as potholes.
For transverse cracks, the performance was degraded on adding Indian data due
to an overall decrease in the proportion of these cracks in total data when
Japanese data is mixed with Indian data with an extremely low number of
transverse cracks ($<1\%$).

For India, adding images from Japan to Indian data, though, did not show much
improvement in performance with respect to empirical analysis, but improves
the generalization capability of the model as demonstrated by the qualitative
analysis. For Czech, adding images from Japan helps in improving the
performance for Linear Cracks. For Alligator cracks and Potholes – the no. of
instances in Czech data are very less, and increasing the proportion of
Japanese data for training the model leads to the dominated effect of
corresponding features learned from Japanese data and ultimately to degraded
performance for Czech.

Overall, it can be concluded that mixed source modeling outperforms single-
source as well as pure modeling in the case of multiple sources. Further, the
following observations can be made:

  1. 1.

Adding data from other countries helps in increasing the generalizability of
the models, as demonstrated by the performance of several Indian, Japanese,
and mixed India-Japan models.

  2. 2.

The effect on improving the performance by adding data from another country
may not be the same/bidirectional for two countries. For instance, mixing
Indian and Japanese data helped in improving the performance for Japanese
roads; however, for Indian roads, only the generalizability of the models was
increased.

  3. 3.

Visual or qualitative analysis is required for accurately assessing the
performance across the countries. For example: For mixed damages (like the
cases when potholes and Alligator Cracks are inter-mixed for India), there can
be multiple possible predictions which are correct, but are marked as
incorrect due to mismatching with the information listed in Ground-Truth xmls.
Qualitative analysis helps in assessing those cases.

The following subsections present the proposed recommendations and discuss the
scope for extending the work in the future.

###  Recommendations

Based on the results and analysis, the study proposes the following
recommendations for readers of other countries when one country, (Japan, in
this case) releases its data and model:

  1. 1.

Can they utilize the Japanese model? Would the performance of the model
inherited from another country when applied to their country remain the same?

     * •

Mostly the performance of a model significantly degrades when applied to road
images from another country (as evident from our experiments).

     * •

Readers should perform qualitative analysis by testing the Japanese model on a
few images from their country. If the performance is not satisfactory, the
readers can follow the steps provided in our work and get a new model for
their country using the Japanese model as the baseline.

  2. 2.

Should they collect images from their country as well? If yes, what effect
would these images have on the performance?

     * •

Yes. The local images help in improving the performance of the model trained
on foreign data.

  3. 3.

Should they mix the images from two countries to train the model or use only
the images from their country (pure modeling vs. mixed modeling)?

     * •

Our research recommends the mixing of images collected from different
countries due to the following reasons:

       1. (a)

The models can automatically detect and classify road damages quickly once
trained using the appropriate data; however, collecting the road images and
preparing the corresponding annotations for Ground-Truth information is a
time-taking task. Mixing the images already available from other countries
helps in increasing the size of the dataset in lesser time.

       2. (b)

It prevents overfitting and improves the generalizability of the model.

       3. (c)

Some damage classes can be better represented using data from other countries
(For example, D10 for India).

       4. (d)

The proposed models are based on transfer learning, and usually, the available
models used as a baseline for transfer learning are trained on any general
dataset, not necessarily related to road damages. If the road damage data or
model is available even though from some other country, using it as the
baseline helps the new model learn the required features for automatic road
damage detection, faster.

###  Future Scope

  1. 1.

This research explores the ways to utilize the data and models available from
one country to design road damage detection and classification models for
other countries. In the future, the same prototype can be extended to propose
a single standardized model that is applicable globally or at least to a set
of countries having identical road conditions.

  2. 2.

Further, only the relative performance of models trained using different
datasets were required to carry out this study. Future work can focus on the
actual performance of the models leading to designing models with improved
accuracy on a similar line.

  3. 3.

Furthermore, this work can be used as a baseline, and the experiments can be
repeated by collecting more images from different countries, including India,
Japan, and Czech, under different seasonal conditions for better representing
each damage class and improving the robustness of the overall detection system
for all damage categories.

  4. 4.

An additional increase in coverage and turnaround time could be achieved by
installing the road damage detection system on smartphones, and car vehicle
recorders mounted in the vehicles operated by municipalities, such as public
transport or waste collection vehicles.

  5. 5.

Finally, public participation through the release of a free smartphone
application used by the public for uploading geo-tagged images of poorly
performing road sections could provide additional data on the current state of
the local road network in several countries.

##  Conclusion

The presented study proposes a dataset of 26620 road images collected from
Japan, India, and the Czech Republic, and analyzes the methods that implement
deep learning for automated detection and classification of road damage. These
methods provide a base for quickly and cost-effectively surveying the road
network by mounting smartphones in the vehicles. A smartphone application
based on such methods has already been designed for Japan and is being used by
several Japanese municipalities for efficient road condition monitoring since
2018. However, several other countries still lack proper research in this
direction. Whether these countries can directly import Japanese or some other
country’s models and application or do they need to make any changes, for
monitoring conditions of their local roads, is still unaddressed. This
research provides recommendations for road agencies of such countries by
demonstrating 30 experiments based on different combinations of data collected
from Japan, India, and Czech. The main conclusion is that the countries can
mix the already available Japanese data with their local road data and can
design an efficient model of their own on a similar line as done in this work.
Additionally, the study proposes Japanese data-based models that can
efficiently detect and classify road damages in India and the Czech Republic.
Thus, the study lays down a foundation for designing a globally applicable
standardized model for road damage detection and classification and is useful
for pavement engineers, road authorities, municipalities, and researchers from
several countries.

## References

  * Adeli, (2001) Adeli, H. (2001).  Neural networks in civil engineering: 1989–2000.  Computer-Aided Civil and Infrastructure Engineering, 16(2):126–142.
  * Alfarrarjeh et al., (2018) Alfarrarjeh, A., Trivedi, D., Kim, S. H., and Shahabi, C. (2018).  A deep learning approach for road damage detection from smartphone images.  In 2018 IEEE International Conference on Big Data (Big Data), pages 5201–5204. IEEE.
  * Anand et al., (2018) Anand, S., Gupta, S., Darbari, V., and Kohli, S. (2018).  Crack-pot: Autonomous road crack and pothole detection.  In 2018 Digital Image Computing: Techniques and Applications (DICTA), pages 1–6. IEEE.
  * Angulo et al., (2019) Angulo, A., Vega-Fernández, J. A., Aguilar-Lobo, L. M., Natraj, S., and Ochoa-Ruiz, G. (2019).  Road damage detection acquisition system based on deep neural networks for physical asset management.  In Mexican International Conference on Artificial Intelligence, pages 3–14. Springer.
  * Budzar, (2018) Budzar, M. (2018).  Re-training the model with images using tensorflow.  Retrieved from: https://proandroiddev.com/re-training-the-model-with-images-using-tensorflow-7758e9eb8db5.
  * Casas-Avellaneda and López-Parra, (2016) Casas-Avellaneda, D. A. and López-Parra, J. F. (2016).  Detection and localization of potholes in roadways using smartphones.  Dyna, 83(195):156–162.
  * Cha et al., (2018) Cha, Y.-J., Choi, W., Suh, G., Mahmoudkhani, S., and Büyüköztürk, O. (2018).  Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types.  Computer-Aided Civil and Infrastructure Engineering, 33(9):731–747.
  * Dertat, (2017) Dertat, A. (2017).  Applied deep learning – part 4: Convolutional neural networks.  Retrieved from: https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2.
  * Douillard, (2018) Douillard, A. (2018).  3 small but powerful convolutional networks.  Retrieved from: https://towardsdatascience.com/3-small-but-powerful-convolutional-networks-27ef86faa42d.
  * Du et al., (2020) Du, Y., Pan, N., Xu, Z., Deng, F., Shen, Y., and Kang, H. (2020).  Pavement distress detection and classification based on yolo network.  International Journal of Pavement Engineering, pages 1–14.
  * Everingham et al., (2010) Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. (2010).  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338.
  * Fan et al., (2018) Fan, Z., Wu, Y., Lu, J., and Li, W. (2018).  Automatic pavement crack detection based on structured prediction with the convolutional neural network.  arXiv preprint arXiv:1802.02208.
  * Girshick, (2015) Girshick, R. (2015).  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448.
  * Girshick et al., (2014) Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014).  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587.
  * Goodfellow et al., (2016) Goodfellow, I., Bengio, Y., and Courville, A. (2016).  Deep learning.  MIT press.
  * Hasenauer et al., (2001) Hasenauer, H., Merkl, D., and Weingartner, M. (2001).  Estimating tree mortality of norway spruce stands with neural networks.  Advances in Environmental Research, 5(4):405–414.
  * He et al., (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
  * Howard et al., (2017) Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017).  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861.
  * Huang et al., (2017) Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al. (2017).  Speed/accuracy trade-offs for modern convolutional object detectors.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7310–7311.
  * Intel, (2018) Intel (2018).  Pedestrian detection using tensorflow on intel architecture (white paper), intel ai builders.  Retrieved from: https://builders.intel.com/docs/aibuilders/pedestrian-detection-using-tensorflow-on-intel-architecture.pdf.
  * IRC:82, (2015) IRC:82 (2015).  Code of practice for maintenance of bituminous road surfaces.  Technical report, Indian Road Congress, New Delhi, India.
  * Kargah-Ostadi et al., (2017) Kargah-Ostadi, N., Nazef, A., Daleiden, J., and Zhou, Y. (2017).  Evaluation framework for automated pavement distress identification and quantification applications.  Transportation Research Record, 2639(1):46–54.
  * Kluger et al., (2018) Kluger, F., Reinders, C., Raetz, K., Schelske, P., Wandt, B., Ackermann, H., and Rosenhahn, B. (2018).  Region-based cycle-consistent data augmentation for object detection.  In 2018 IEEE International Conference on Big Data (Big Data), pages 5205–5211. IEEE.
  * Lin et al., (2017) Lin, Y.-z., Nie, Z.-h., and Ma, H.-w. (2017).  Structural damage detection with automatic feature-extraction through deep learning.  Computer-Aided Civil and Infrastructure Engineering, 32(12):1025–1046.
  * Liu et al., (2016) Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A. C. (2016).  Ssd: Single shot multibox detector.  In European conference on computer vision, pages 21–37. Springer.
  * Maeda et al., (2020) Maeda, H., Kashiyama, T., Sekimoto, Y., Seto, T., and Omata, H. (2020).  Generative adversarial network for road damage detection.  Computer-Aided Civil and Infrastructure Engineering.
  * Maeda et al., (2018) Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T., and Omata, H. (2018).  Road damage detection and classification using deep neural networks with smartphone images.  Computer-Aided Civil and Infrastructure Engineering, 33(12):1127–1141.
  * Majidifard et al., (2020) Majidifard, H., Jin, P., Adu-Gyamfi, Y., and Buttlar, W. G. (2020).  Pavement image datasets: A new benchmark dataset to classify and densify pavement distresses.  Transportation Research Record, 2674(2):328–339.
  * McGhee, (2004) McGhee, K. (2004).  Automated pavement distress collection techniques. nchrp synthesis 334\. national cooperative highway research program.  Transportation Research Board, Washington DC.
  * Mertz et al., (2014) Mertz, C., Varadharajan, S., Jose, S., Sharma, K., Wander, L., and Wang, J. (2014).  City-wide road distress monitoring with smartphones.  In Proceedings of ITS World Congress, pages 1–9.
  * Miller and Zaloshnja, (2009) Miller, T. R. and Zaloshnja, E. (2009).  On a crash course: The dangers and health costs of deficient roadways.
  * Peppa et al., (2018) Peppa, M., Bell, D., Komar, T., and Xiao, W. (2018).  Urban traffic flow analysis based on deep learning car detection from cctv image series.  In SPRS TC IV Mid-term Symposium “3D Spatial Information Science–The Engine of Change”. Newcastle University.
  * Pierce et al., (2013) Pierce, L. M., McGovern, G., and Zimmerman, K. A. (2013).  Practical guide for quality management of pavement condition data collection.
  * Prabhu, (2018) Prabhu, R. (2018).  Understanding of convolutional neural network (cnn)–deep learning.  A Medium Corporation, US.
  * Radopoulou and Brilakis, (2015) Radopoulou, S. C. and Brilakis, I. (2015).  Detection of multiple road defects for pavement condition assessment.  Eindhoven, The Netherlands.
  * Ren et al., (2015) Ren, S., He, K., Girshick, R., and Sun, J. (2015).  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99.
  * Roadware, (2019) Roadware, F. (2019).  Pavement condition assessment, data sheets.  Retrieved from: http://www.fugroroadware.com/related/english-alldatasheets.
  * Roberts et al., (2020) Roberts, R., Giancontieri, G., Inzerillo, L., and Di Mino, G. (2020).  Towards low-cost pavement condition health monitoring and analysis using deep learning.  Applied Sciences, 10(1):319.
  * Sharma, (2017) Sharma, S. (2017).  Activation functions: Neural networks.  Retrieved from: https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6.
  * Silva and Lucena, (2018) Silva, W. R. L. d. and Lucena, D. S. d. (2018).  Concrete cracks detection based on deep learning image classification.  In Multidisciplinary Digital Publishing Institute Proceedings, volume 2, page 489.
  * Szegedy et al., (2016) Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016).  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826.
  * (42) Wang, W., Wu, B., Yang, S., and Wang, Z. (2018a).  Road damage detection and classification with faster r-cnn.  In 2018 IEEE International Conference on Big Data (Big Data), pages 5220–5223. IEEE.
  * (43) Wang, Y. J., Ding, M., Kan, S., Zhang, S., and Lu, C. (2018b).  Deep proposal and detection networks for road damage detection and classification.  In 2018 IEEE International Conference on Big Data (Big Data), pages 5224–5227. IEEE.
  * Zalama et al., (2014) Zalama, E., Gómez-García-Bermejo, J., Medina, R., and Llamas, J. (2014).  Road crack detection using visual features extracted by gabor filters.  Computer-Aided Civil and Infrastructure Engineering, 29(5):342–358.
  * Zhang et al., (2017) Zhang, A., Wang, K. C., Li, B., Yang, E., Dai, X., Peng, Y., Fei, Y., Liu, Y., Li, J. Q., and Chen, C. (2017).  Automated pixel-level pavement crack detection on 3d asphalt surfaces using a deep-learning network.  Computer-Aided Civil and Infrastructure Engineering, 32(10):805–819.
  * Zhang et al., (2016) Zhang, L., Yang, F., Zhang, Y. D., and Zhu, Y. J. (2016).  Road crack detection using deep convolutional neural network.  In Image Processing (ICIP), 2016 IEEE International Conference on, pages 3708–3712. IEEE.