# Crowdsensing-based Road Damage Detection Challenge (CRDDC’2022)

Deeksha Arya⋆5, Hiroya Maeda3, Sanjay Kumar Ghosh4, Durga Toshniwal2, Hiroshi
Omata5,
Takehiro Kashiyama8, Yoshihide Sekimoto5
<EMAIL_ADDRESS>
5 Centre for Spatial Information Science, The University of Tokyo, 4-6-1
Komaba, Tokyo, Japan
3 UrbanX Technologies, Inc., Tokyo, Japan
4 Department of Civil Engineering, Indian Institute of Technology Roorkee,
247667, India
2 Department of Computer Science and Engineering, Indian Institute of
Technology Roorkee, 247667, India
8 Faculty of Economics, Osaka University of Economics, Japan

###### Abstract

This paper summarizes the Crowdsensing-based Road Damage Detection Challenge
(CRDDC), a Big Data Cup organized as a part of the IEEE International
Conference on Big Data’2022. The Big Data Cup challenges involve a released
dataset and a well-defined problem with clear evaluation metrics. The
challenges run on a data competition platform that maintains a real-time
online evaluation system for the participants. In the presented case, the data
constitute 47,420 road images collected from India, Japan, the Czech Republic,
Norway, the United States, and China to propose methods for automatically
detecting road damages in these countries. More than 60 teams from 19
countries registered for this competition. The submitted solutions were
evaluated using five leaderboards based on performance for unseen test images
from the aforementioned six countries. This paper encapsulates the top 11
solutions proposed by these teams. The best-performing model utilizes ensemble
learning based on YOLO and Faster-RCNN series models to yield an F1 score of
76% for test data combined from all 6 countries. The paper concludes with a
comparison of current and past challenges and provides direction for the
future.

###### Index Terms:

Object Detection, Classification, Road Maintenance, Intelligent Transport, Big
Data, Deep Learning, Database, Smart City Applications.

##  I Introduction

Road safety continues to be a pressing concern, with road traffic accidents
being a global disaster hampering the life of millions [1, 2, 3]. Poor road
conditions are one of the major factors leading to road accidents [4, 5].
Apropos, there is an immediate need for fresh inventions and an upward
technological trajectory to reform the road and transportation arena. The
paper addresses the domain of AI (Artificial Intelligence)-driven road
condition inspection [4] through the Crowdsensing-based Road Damage Detection
challenge $(CRDDC$’2022). The background information for CRDDC is provided as
follows.
A lack of financial resources makes many local governments unable to conduct
sufficient road condition inspections on time [6]. Some municipalities
automate road damage detection by using high-performance sensors.
Nevertheless, the high cost of these sensors makes it infeasible to use them
at the country level owing to the vast area of roads to be inspected. This
leads to the need of a system that makes it easy to assess road conditions and
identify the damage to the road surface at a low cost. Recognizing this need,
the first Road Damage Detection Challenge, RDDC’2018, was organized as an IEEE
Big Data Cup in 2018 to evaluate the contemporary methods working towards the
same goal[7, 8, 9, 10, 11, 12, 13, 14]. After the challenge, several
municipalities in Japan started utilizing the proposed automatic road damage
detection system [15]. The practical use and the feedback of government
agencies suggested that the algorithms need to be more robust. Further, it was
observed that most of the existing models are limited to road conditions in a
single country. The development of a method that applies to more than one
country leads to the possibility of designing a stand-alone system for road
damage detection worldwide.
Considering the requirement, Arya et al.[6] augmented the Japanese dataset
with road damage images from India and the Czech Republic. The authors
proposed the data, Road Damage Dataset-2020 (RDD2020 [16, 17]), which
comprises 26620 images, almost thrice the volume of the 2018 dataset[18]
utilized for $RDDC^{\prime}2018$. Further, the authors[6] conducted a
comprehensive analysis of models trained using different combinations of the
data.

This RDD2020 data was made publicly available and formed the basis for
organizing the second challenge of the road damage detection challenge series,
named the Global Road Damage Detection Challenge, GRDDC’2020 [19]. It invited
the participants to propose models capable of efficiently detecting road
damage in India, Japan, and the Czech Republic. GRDDC received an overwhelming
response from the research community with a participation of 121 teams([20,
21, 22, 23, 24, 25, 26, 27, 28]).

Considering the wide impact of RDDC’2018 and GRDDC’2020, the challenge CRDDC
has been brought forward in 2022. For RDDC’2018 and GRDDC’2020, the dataset
was provided solely by the challenge organizers, and participants were
restricted to using only the challenge dataset to train the models. However,
the CRDDC’2022, allows the participants to develop/propose their datasets,
eliminating the constraint of RDDC’2018 and GRDDC’2020. Thus, the central
theme of the challenge is “Crowdsensing,” – which refers to the crowdsourcing
of sensor data collected as per the choice of contributors. It may be noted
that, even though CRDDC allowed participant to propose their own dataset, it
is different than the data-centric challenge organized by Behzadian et
al.[29]. Behzadian et al. asked the participants to increase the accuracy of a
predefined model architecture by utilizing various data modification methods
such as cleaning, labeling and augmentation. The current challenge CRDDC’2022
is different in the following ways:

  1. 1.

The model architecture is not predefined. Participants may propose any
architecture.

  2. 2.

The data modification methods such as cleaning and augmentation are allowed to
increase the accuracy of the proposed model in Phase 3 of the challenge.
However, adding new data or labels is allowed only in Phase 1 of the
challenge.

Anyone having the dataset could register for CRDDC as a data contributor, data
recommender, or information provider by participating in the first phase of
the challenge. Phase 2 involved a suitability analysis to select the datasets
for releasing through the CRDDC’2022, and utilization for Phase 3 tasks. This
paper presents a summary of this challenge, along with the solutions proposed
by the participants during various phases.

##  II Challenge Review

###  II-A Overview

The CRDD challenge is designed to push the state-of-the-art in detecting road
damages forward. The challenge comprises two components: (i) a publicly
available dataset of road images and annotation and (ii) an online competition
and workshop. The CRDDC’2022 dataset consists of annotated road damage images
collected from India, Japan, the Czech Republic, Norway, the United States,
and China. There are two key challenges: classification – “does the image
contain any instances of a particular road damage class (where the road damage
classes include longitudinal cracks, transverse cracks, potholes, etc.)?”, and
detection – “where are the instances of a particular road damage class in the
image (if any)?” The data challenges are issued with deadlines each year, and
a workshop is held to compare and discuss that year’s results and methods. The
datasets and associated annotation and evaluation methods are subsequently
released and made available for use at any time.

The procedure for the previous challenge, GRDDC’2020, was inspired by the
PASCAL VOC challenge that was organized from 2005 to 2012 (see [30, 31]). The
procedure for CRDDC’2022 is similar to GRDDC; however, it includes some extra
parts like the data contributed by the participants, new leaderboards, etc.
Also, like GRDDC, the objectives of the CRDDC are twofold: first to provide
challenging road damage images and high-quality annotation, together with a
standard evaluation methodology to compare the performance of algorithms
fairly (the dataset component); and second, to measure state of the art for
the current year (the competition component). This paper aims to describe the
challenge, the methods used, the evaluation carried out, and the results. The
comparative analysis of three challenges conducted till 2022 is also covered.

###  II-B Dataset

The data released through CRDDC is named RDD2022 and is publicly available
[32]. It constitutes road images collected from India, Japan, the Czech
Republic, Norway, the United States, and China. Our companion paper [33]
presents the details of the data, including the data collection methodology,
the study area, the annotation procedure, etc. These details are not repeated
here. The challenge divides the data in two chunks: Train and Test. The set
“Train” consists of road images with annotations in XML files in PASCAL VOC
format. The “Test” set is released without the annotations for evaluating the
solutions proposed by the participants. The participants need to predict the
annotation for the test images using the proposed model. The predicted
annotations are evaluated by an online server accessible on the challenge
website. The evaluation procedure is described in the next section. The
statistics for the distribution of images in the six countries are provided in
figure 1. The training data contains annotation with marked labels for four
types of road damage. Figure 2 shows the number of instances for each damage
type in training data. The reason for considering these four damage types is
the same as provided in [6].

Figure 1: Distribution of images across different countries in RDD2022 Figure
2: Distribution of different damage type instances in training data released
for CRDDC’2022

###  II-C Tasks

The challenge was organized in four phases with different tasks for the
participants. The details are as follows.

  1. 1.

Phase 1: Data Solicitation: The participants were asked to submit a report of
their proposed road damage dataset (cracks, potholes, or any other damage
categories) by providing the details through the corresponding questionnaire.
The following categories of participants were considered:

    1. (a)

Data Contributors (Data owners interested in contributing their data to CRDDC)

    2. (b)

Information Providers (Data owners interested to participate by providing only
the information related to their dataset, keeping the data private)

    3. (c)

Data Recommenders (Researchers, Academicians, Freelancers, interested in
recommending the inclusion of others’ datasets to CRDDC)

  2. 2.

Phase 2: Data Shortlisting: Based on the reports submitted in phase 1, the
shortlisted teams were asked to submit a link to their proposed dataset in
Phase 2. After verification, the winners (data contributors) of phase 1 were
announced. The solicited datasets were added to the Challenge datasets for the
next phase of the challenge.

  3. 3.

Phase 3: Main Task: The participants were asked to propose road damage
detection models targeting the following 5 cases using any combination of the
CRDDC dataset:

    1. (a)

Models applicable to all six countries (Overall category)

    2. (b)

Models applicable to India

    3. (c)

Models applicable to Japan

    4. (d)

Models applicable to the United States

    5. (e)

Models applicable to Norway

The participants were allowed to propose models targeting one or multiple
countries. For instance - Model A for Country 1, Model B for Country 2, Model
C for Countries 3 and 4, or a single model for all countries could be
proposed.

The LeaderBoard-1 (Overall category for all 6 countries) required the
participants to propose an algorithm that automatically recognizes the road
damages in an image captured from any of the underlying six countries. The
recognition implies detecting the damage location in the image and identifying
the damage type. Using this algorithm, the participants need to predict labels
and the corresponding boundary boxes for damage instances present in each test
data image. The predictions need to be submitted as a CSV file through the
submission link provided on the website. The general rules to be followed
while accomplishing the challenge tasks are stated below:

    1. (a)

Algorithm: Recently, Deep Learning models have become the first choice for
applications involving object detection. However, the challenge allows all
types of algorithms as long as the proposed solution can outperform other
solutions.

    2. (b)

Pre-trained Models: The challenge permits the use of transfer learning
utilizing pre-trained models.

    3. (c)

Data: Participants are restricted to train their algorithms on the provided
Road Data train sets. Collecting additional data for the target attribute
labels is not allowed. However, it is permitted to increase training images
using data augmentation, GANs, etc., artificially.

    4. (d)

License: The dataset released for the challenge is publicly available under
the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-
SA 4.0).

  4. 4.

Phase 4: Report and Source Code: Phase 4 required the participants to submit a
report of their proposed solution along with the source code.

The following section provides details about the submission and evaluation of
the results.

##  III Submission and Evaluation

###  III-A Submission of Results

The running of the challenge consisted of four phases with submission at
multiple stages, as described below.

  1. 1.

At the start of the challenge, the task for Phase 1 and Phase 2 were
announced, and participants were asked to submit the information of the
dataset they wanted to include for the $CRDDC$-Phase 3. The information was
collected through a set of questionnaires released as Google Forms and the
submission of a report for the proposed dataset.

  2. 2.

In phase 2, the shortlisted participants of Phase 1 were asked to provide a
link to their proposed dataset. Both images and annotations were required at
this stage.

  3. 3.

In phase 3, the $CRDDC$ dataset, RDD2022, comprising RDD2020 extended with the
contribution of Phase 1 winners, was released. The dataset details are
provided in our companion paper [32]. Unlike $GRDDC$’2020, both Train and Test
sets were released at once in $CRDDC$. The Train set included images and
annotations, whereas the Test set included only images with no annotation
files. Participants were required to train their models using the Train set,
infer the results for Test data and submit results to an evaluation server.
The server was designed with an evaluation script, which calculates a score
corresponding to every submission. The calculated score is added to the
private leaderboard of the participants. Simultaneously, a public leaderboard
is maintained on the challenge website (https://crddc2022.sekilab.global/) to
report the highest score achieved by the teams on their private leaderboards.
Five leaderboards for each category (private and public) were maintained to
cover the five target cases mentioned in Phase 3 tasks.

  4. 4.

In the final phase, the participants were required to submit their source code
to verify the replicability of the proposed algorithms.

The evaluation procedure for the submissions is described in the following
sub-section.

###  III-B Evaluation

The evaluation procedure for $CRDDC$’2022 is same as that of $GRDDC$’2020,
except that there are five concurrent leaderboards for $CRDDC$, unlike
$GRDDC$, which had two leaderboards for different test sets. The results
submitted by the participants are evaluated using an evaluation script
embedded in the server. The script utilizes two inputs: (i) the ground truth
information of the test dataset, the annotations, and (ii) the file containing
the participants’ predictions. A prediction is considered correct if it
satisfies the following two criteria: The area of overlap between the
predicted bounding box and ground truth bounding box exceeds 50$\%$. The
condition is described as $IoU>0.5$. $IoU$ denotes Intersection Over Union.
For more details regarding this, please refer to [34]. The predicted label
matches the actual label, as specified in the annotation file (ground truth)
of the image. The script compares the two input files and calculates F1-Score
for the submission. The F1-score is calculated as the Harmonic Mean of
Precision and Recall values. Precision is the ratio of true positives to all
predicted positive. The recall is the ratio of true positives to all actual
positives. The details of the parameters are given below:

  1. 1.

True Positive (TP): When a damage instance is present in the ground truth, and
the label and the bounding box of the instance are correctly predicted with
$IoU>0.5$.

  2. 2.

False Positive (FP): When the model predicts a damage instance at a particular
location in the image, but the instance is not present in the ground truth for
the image. This also covers the case when the predicted label doesn’t match
with the actual label.

  3. 3.

False Negative (FN): When a damage instance is present in the ground truth,
but the model fails to predict either the correct label or the bounding box of
the instance.

  4. 4.

Recall:

$Recall={\frac{(TP)}{(TP+FN)}}$ (1)

  5. 5.

Precision:

$Precision={\frac{(TP)}{(TP+FP)}}$ (2)

  6. 6.

F1-Score:

$F1-Score=2\times{\frac{(Precision\times Recall)}{(Precision+Recall)}}$ (3)

The F1 metric weights recall and precision equally. Thus, the competition
favors moderately good performance on both over the outstanding performance on
one and poor performance on the other. F1 is calculated separately for each of
the five target cases defined in Phase 3 tasks. Finally, the average of all
five scores is used to rank the teams.

##  IV CRDDC’2022: Results and Rankings

TABLE I: Details of the winners of Data Contribution Phase for CRDDC’2022 (Phase 1 and 2) Rank |  Team Name |  Contributors |  Affiliation | Contribution
---|---|---|---|---
1 |  NTNU |  Madhavendra Sharma, Mamoona Birkhez Shami, Muneer Al-Hammadi, Alex Klein-Paste, Helge Mork, and Frank Lindseth, |  Norwegian University of Science and Technology, Norway | Data from Norway
2 |  AGRD |  Van Vung Pham, Du Nguyen |  Sam Houston State University, United States | Data from the United States
3 |  Futuristic Detection |  Jingtao Zhong, Hanglin Cheng, Jing Zhang |  Southeast University, China | Data from China
TABLE II: CRDDC Ranks and Scores of top 11 teams (Phase 3 and 4) Rank | Team Name |  | LeaderBoard - 1
---
(overall 6 countries)
| LeaderBoard - 2
---
(India)
| LeaderBoard - 3
---
(Japan)
| LeaderBoard - 4
---
(Norway)
| LeaderBoard - 5
---
(United States)
Average Score
1 | ShiYu_SeaView | 0.770 | 0.583 | 0.789 | 0.595 | 0.844 | 0.716
2 | DongjunJeong | 0.743 | 0.540 | 0.750 | 0.538 | 0.801 | 0.674
3 | MDPT | 0.741 | 0.516 | 0.735 | 0.504 | 0.817 | 0.663
4 | SGG-RS-Group | 0.727 | 0.545 | 0.727 | 0.481 | 0.779 | 0.652
5 | IRCV-URV | 0.726 | 0.518 | 0.735 | 0.498 | 0.779 | 0.651
6 | IMSC | 0.728 | 0.542 | 0.725 | 0.478 | 0.774 | 0.649
7 | NJUPT | 0.718 | 0.559 | 0.720 | 0.458 | 0.743 | 0.640
8 | TUT | 0.694 | 0.519 | 0.773 | 0.464 | 0.727 | 0.636
9 | MILA | 0.697 | 0.494 | 0.716 | 0.462 | 0.775 | 0.629
10 | SIAI | 0.612 | 0.417 | 0.608 | 0.424 | 0.663 | 0.545
11 | kubapok | 0.603 | 0.421 | 0.594 | 0.383 | 0.649 | 0.530
TABLE III: Details of the Contributors (Top 11 teams for CRDDC’2022 - Phase 3 and 4) Team Name |  Contributors |  Affiliation | Country
---|---|---|---
ShiYu_SeaView |  Wenchao Ding, Xu Zhao, Bingke Zhu, Yinglong Du, Guibo Zhu, Tao Yu, Lei Li, and Jinqiao Wang |  Objecteye Inc. | China
DongjunJeong |  (1) Dongjun Jeong, ((2) Jua Kim |  (1) OPGG, (2) X (freelancer) | Republic of Korea
MDPT |  Vung Pham, Du Nguyen, Christopher Donan |  Sam Houston State University | United States
SGG-RS-Group |  Yao Tang, Xusi Liao, Jiang He, Haoliang Feng, Hongzan Jiao, Xin Su, Qiangqiang Yuan |  Wuhan University | China
IRCV-URV |  Ammar M. Okran, Mohamed Abdel-Nasser, Hatem A. Rashwan, Domenec Puig |  Universitat Rovira i Virgili | Spain
IMSC |  Seon Ho Kim, Maitry Bhavsar, Utkarsh Baranwal, Abdullah Alfarrarjeh |  University of Southern California (USA), Indian Institute of Technology Patna (India), German Jordanian University (Jordan) | USA, India, Jordan
NJUPT |  Shihao Han, Guoping Jiang, Yingjiang Zhou, Haowen Xu, Jiajing Ying, Huan Yang |  Nanjing University of Posts and Telecommunications, College of Automation and Artificial Intelligence | China
TUT |  Lu Yang, Hualin He, Tao Liu |  Tianjin University of Technology | China
MILA |  Poonam Kumari Saha, Yoshihide Sekimoto |  Sekimoto Lab, The University of Tokyo | Japan
SIAI |  Jeon Yun Tae, Dai Quoc Tran |  Department of Global Smart City, Sungkyunkwan University | Korea
kubapok |  Jakub Pokrywka |  Adam Mickiewicz University | Poland
TABLE IV: Details of the solutions proposed by top 11 teams for CRDDC’2022 - Phase 3 and 4 Team Name |  Data utilized |  Proposed Solution | Ensemble learning |  Main Feature
---|---|---|---|---
ShiYu_SeaView |  RDD2022 |  Ensemble model based on YOLO-series and Faster RCNN-series models | Yes |  Multi-model ensemble: 1.YOLO-series models to achieve Dense predictions, and 2. the Faster RCNN-series model with Swin Transformer as backbone to achieve more accurate predictions by their large receptive fields and two-stage methods.
DongjunJeong |  RDD2022 |  YOLOv5x P5 and P6 Ensemble with Image patch | Yes |  1\. Use of Image patch strategy to deal with high-resolution images. and 2. Use of pre-trained weights based on GRDDC’2020 winning solution.
MDPT |  RDD2022 |  YOLOv7 with train and test image augmentations,label smoothing, and coordinate attentions | No |  1\. Cropping the Norway images to focus only on road areas and support training with limited hardware resources. 2. Training country-specific models using data from all countries by varying the images used for validation.
IMSC |  RDD2022 |  YOLOv5 based ensemble | Yes |  Use of customized Anchor Boxes tuned according to the used training dataset to fit the sizes of objects of interest.
SGG-RS-Group |  RDD2022 except China_Drone |  The ensemble model based on YOLOv5 and attention modules | Yes |  Use of Attention modules to enhance important feature maps, such as highlighting road regions in the image.
IRCV-URV |  RDD2022 |  Yolov7-based ensemble | Yes |  Experiments to ensemble models trained with different image-size and architectures, including the use of Atrous Spatial Pyramid Pooling.
NJUPT |  RDD2022 except China_Drone |  yolov7x, yolov5x, yolov5x-transformer | Yes |  Experiments to ensemble models based on the input image size, attention mechanisms, and data augmentations. Shows that more models don’t always mean higher accuracy.
MILA |  RDD2022 except China_Drone (Only positive images) |  YOLOv7-based ensemble | Yes |  1\. Innovative Data Augmentation based on 17 techniques. 2. Consideration of only Positive samples to support faster training through image caching.
TUT |  RDD2022 |  YPLNet (Yolov5s + Pyramid Squeeze Attention + Large-field Contextual Feature Integration) | No |  Extraction and fusion of multi-scale contextual features using PSA and LCFI with limited computational cost.
SIAI |  RDD2022 except China_Drone |  YOLOv5 + VFNet in MMDetection | Yes |  1\. Image scaling applied to Norway images. 2. VFNet worked well with rectangular images and Yolov5 for square.
kubapok |  RDD2022 |  YOLOv5 based ensemble | Yes |  Optimal GPU time usage by an ensemble of models trained with multiple levels of augmentations.

More than 50 teams from 19 different nationalities registered for the
challenge. The related data is available on the challenge website
(https://crddc2022.sekilab.global/participants/). The details of shortlisted
solutions and winners of different phases are provided as follows.

###  IV-A Phase 1 and 2: Data Solicitation

For the data contribution phase, the datasets submitted by the three teams
listed in Table I, were found the most suitable for merging with the existing
RDD2020 dataset. The combined dataset was released as RDD2022 and provided to
Phase 3 participants. The RDD2022 data has been made freely available at the
FigShare repository [32]. The details are provided in [33].

###  IV-B Phase 3 and 4: Model Contribution

The top 11 teams were shortlisted based on the average F1 score achieved by
the proposed solutions, the submissions of source code, and a detailed report.
Table II lists the ranking achieved by these teams and their scores
corresponding to the five leaderboards. The details of the contributors are
presented in tables III. The top 5 teams achieved an average F1 score of more
than 65% and proposed efficient methods using ensemble learning and several
data augmentation techniques. Table IV summarizes the details of the top 11
solutions proposed targeting the six countries together (Overall_6_countries
leaderboard).

It may be noted that, the state-of-the-art object detectors are grouped into
two categories: two-stage detector[35, 36, 37, 38] and one-stage detector [39,
40, 41, 42, 43, 44, 45, 46]. As the name suggests, the two-stage detector
works in two steps: it first extract proposals and then send them into the
later network for fine-grained classification and location. Whereas, the one-
stage detector performs these two tasks in one step - the images are fed into
the network to get the result directly. As a result, one-stage detectors are
usually faster.

The winning method for CRDDC’2022, proposed by team ShiYu_SeaView, is based on
an ensemble of one-stage and two-stage detectors. For one-stage, the team
adopted YOLOv5[45] and YOLOv7[46]. YOLOv5[47] is choosen because it showed
high performance in solutions proposed through GRDDC’2020[19] targeting the
same domain of road damage detection. The YOLOv7[46] is selected because it
has several trainable bag-of-freebies, e.g., model re-parametrization, and
auxiliary head, and is a new state-of-the-art real-time object detector.

Likewise, the team adopted Cascade RCNN[38] models for two-stage detectors
after analyzing the top-ranking solutions[20, 21, 22] for GRDDC’2020[19]. From
the anlaysis, the team inferred that YOLO series models perfer to have a
higher recall rate, while Cascade RCNN series models prefer to have a higher
precision rate and works better for small damage. Based on the inference,
ensemble learning of these two types of models was proposed to have
complementary positive effects on the performance.

Further, the team also utilized Swin Transformer[48, 49] as the Backbone and
Deformable ROI Pooling as the ROI head of the two-stage detector. This is
because the self-attention module (transformer[50]) enables better extraction
of semantic information and has a larger receptive field. That is considered
useful for the current problem domain because cracks often have very long and
narrow morphologies and traditional $3\times 3$ convolution kernels may not
extract these features properly. Deformable convolution[51, 52] sets the
learnable offsets to solve geometric transformation problems. Besides, the ROI
Pooling[51] can better adapt to the special shape of road damage.

Similar to the Rank 1 team, ShiYu_SeaView, the team ranked second in CRDDC,
Dongjun Jeong, utilized the GRDDC’2020 winning solutions, to decide their
approach for CRDDC’2022. The proposed approach include the use of transfer
learning using the COCO dataset pre-trained weight and the pre-trained weight
of GRDDC’2020 winner USC-InfoLab (Team IMSC [20]).

This shows the contribution made by the road damage detection challenges and
participants to state-of-the-art for the domain over the years.

##  V Discussion and Future Scope

TABLE V: Comparison of Road Damage Detection Challenges from 2018 to 2022 |  RDDC’2018 |  GRDDC’2020 |  CRDDC’2022
---|---|---|---
#Countries targeted |  1 |  3 |  6
#Teams |  59 |  121 |  64 (at the time of writing)
Difficulty level |  High (First challenge of the series) |  Higher (mainly due to the geographical diversity of target countries) |  Highest (Geographical diversity + multi-resolution images), multiple sub-tasks
Main task |  Model for only Japan |  Combined model for 3 countries |  More challenging, combined model for 6 countries, an individual model for 4 countries
Damage categories |  8 |  4 |  4
Positive points |  (i) Standard Method of Assessment, (ii) Fixing the test data for evaluating the performance |  (i) Foundation for proposing multi-country solutions, (ii) Availability of heterogeneous data to train more robust models. |  (i) Heterogeneity in the data improved - multi-sensor, multi-country, multi-resolution images, (ii) Data contribution by participants allowed, (iii) Standard Method of Assessment for multiple target use cases, (iv) More than 5500 labels provided for each class.
Type of challenge |  Model-centric, but participants allowed to augment the data artificially. |  Model-centric, but participants allowed to augment the data artificially. |  Model-centric, with more freedom for the participants to experiment with the data. Artificial augmentation - allowed. Additionally, Phase 1 invited participants to contribute their data.
Test data |  Two sets - Test 1 and Test2, released at different times, comprising images from Japan. |  Two sets - Test 1 and Test2, released at different times, comprising images from Japan, India, and Czech Republic. |  One set - Test, comprising images from underlying 6 countries, released at once together with training data.
Time for training the models |  Around 4 months before the test1 deadline |  Around 4 months before the test1 deadline |  Around 2 months
Number of leaderboards |  Two with different deadlines (Test 1 and Test2) |  Two with different deadlines (Test 1 and Test2) |  Five with same deadline (Overall_6_countries, India, Japan, Norway, United States)
Ranking Criteria |  Average score for test1 and test2 |  Average score for test1 and test2 |  Average score for all 5 leaderboards
Best score achieved |  0.665 (for Japan) |  0.68 (for India, Japan, and Czech Republic) |  0.716 (average for 5 leaderboards), 0.769 (for all 6 countries combined), 0.780 (for Japan)
Best performing model |  Ensemble of Faster-RCNN and SSD, with ImageNet pre-trained ResNet-101 and VGG-16 for the backbone models, respectively |  YOLOv5-based ensemble model |  Ensemble model based on YOLO-series and Faster RCNN-series models

The significance of Road Damage Detection challenges lies in providing a rich
dataset and a standardized evaluation framework for comparing different
methods. The CRDD challenge has lived up to the mark of these expectations.
Table 5 presents a comparative summary of the three road damage detection
challenges organized till now concerning multiple factors. Clearly, CRDDC’2022
made significant contributions to the existing literature by consideration of
multiple use cases. Following are some directions that may be considered for
future road damage detection challenges.

  1. 1.

Due to the consideration of multiple phases in CRDDC’2022, the participants
got lesser time to perform experiments and train the models (Phase 3). Some
participants requested to reopen the leaderboards. Even though new
leaderboards on the challenge website were created to support more experiments
by CRDDC participants and users who couldn’t join the competition, future
challenges may plan and allocate more time for the competition tasks.

  2. 2.

Inclusion of high-resolution images in the dataset RDD2022, released through
CRDDC’2022, led to the requirement of heavy-computation resources to process
the data. This restricts the participation of users with limited resources.
Future challenges may address this issue by providing support for
computational resources, such as Graphics Processing Units (GPU), to
participants in need.

  3. 3.

Though CRDDC allowed participants to propose and contribute the data of their
choice in Phase 1, the participants were restricted to utilizing only the
provided training data in Phase III. In the future, the utilization of other
data sources may also be allowed.

  4. 4.

Until now, $RDD$ challenges focussed only on the detection and classification
of road damages. Future challenges may introduce new tasks such as Severity
analysis, Pixel-level damage analysis for covering a wider spectrum for road
condition monitoring.

  5. 5.

Further, multiple evaluation metrics can be introduced for assessing the
performance better. For instance, considering the inference speed and size of
the proposed model in addition to F1-Score would be useful for developing
real-time smartphone-based object detection methods.

  6. 6.

Furthermore, data balancing at multiple levels may be considered for the
upcoming years concerning countries, damage categories, etc.

  7. 7.

Additionally, the complexity level for CRDDC was higher than that of previous
challenges on various factors listed in Table V. Simpler challenges may be
considered in the future to analyze the performance of the same model for
different countries, keeping the data collection process (mechanism, image
capturing device, image resolution, etc.) uniform.

##  VI Conclusion

The IEEE BigData cup, Crowdsensing-based Road Damage Detection challenge,
focuses on the detection and classification of road damage in different parts
of the world, particularly India, Japan, the Czech Republic, Norway, the
United States, and China. It includes four common damage categories:
Longitudinal Cracks, Transverse Cracks, Alligator Cracks, and Potholes. Since
the data comes from different regions of the world, including developed and
developing countries, the quality of road hardening varies. Further, the
degree of damage, area, aspect ratio, etc., of each type of damage differs
significantly. Furthermore, the data has been captured using multiple
mechanisms – including Smartphone-mounted vehicles, wide-angle cameras,
drones, motorbikes etc., with different image resolutions. Therefore, the
images have high levels of variation in intensity and texture content. Thus,
the BigData cup offers a major challenge to the participants to explore the
most relevant techniques.

This paper summarizes the challenge, elaborating on the main characteristics,
the complexities of the tasks to solve, and the overall evaluation
methodology. It also describes the main features of the approaches proposed by
the highest-ranked contestants and proposes recommendations for future
challenges. In a nutshell, the contest and the summary paper may be used as a
benchmark for the state of the art since multiple solutions are presented with
valuable insights and contribution from the research community, and the
dataset is open for future testing.

## Acknowledgment

We thank all the participants of the challenge for their contributions. We
also thank the sponsor UrbanX Technologies for providing the requisite funds.

## References

  * [1] M. Haghani, A. Behnood, V. Dixit, and O. Oviedo-Trespalacios, “Road safety research in the context of low-and middle-income countries: Macro-scale literature analyses, trends, knowledge gaps and challenges,” _Safety science_ , vol. 146, p. 105513, 2022.
  * [2] M. Tavakkoli, Z. Torkashvand-Khah, G. Fink, A. Takian, N. Kuenzli, D. de Savigny, and D. Cobos Muñoz, “Evidence from the decade of action for road safety: a systematic review of the effectiveness of interventions in low and middle-income countries,” _Public health reviews_ , p. 2, 2022.
  * [3] J. Shah, K. Bhatt, P. Trivedi, and S. Easa, “Road safety conditions and management in india: Challenges and opportunities,” _Infrastructure Planning and Management in India_ , pp. 3–23, 2022.
  * [4] S. Bhattacharya, H. Jha, and R. P. Nanda, “Application of iot and artificial intelligence in road safety,” _2022 Interdisciplinary Research in Technology and Management (IRTM)_ , pp. 1–6, 2022.
  * [5] M. D. Jakobsen, K. Glies Vincents Seeberg, M. Møller, P. Kines, P. Jørgensen, L. Malchow-Møller, A. B. Andersen, and L. L. Andersen, “Influence of occupational risk factors for road traffic crashes among professional drivers: systematic review,” _Transport Reviews_ , pp. 1–31, 2022.
  * [6] D. Arya, H. Maeda, S. K. Ghosh, D. Toshniwal, A. Mraz, T. Kashiyama, and Y. Sekimoto, “Deep learning-based road damage detection and classification for multiple countries,” _Automation in Construction_ , vol. 132, p. 103935, 2021.
  * [7] L. Ale, N. Zhang, and L. Li, “Road damage detection using retinanet,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 5197–5200.
  * [8] A. Alfarrarjeh, D. Trivedi, S. H. Kim, and C. Shahabi, “A deep learning approach for road damage detection from smartphone images,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 5201–5204.
  * [9] F. Kluger, C. Reinders, K. Raetz, P. Schelske, B. Wandt, H. Ackermann, and B. Rosenhahn, “Region-based cycle-consistent data augmentation for object detection,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 5205–5211.
  * [10] V. Mandal, L. Uong, and Y. Adu-Gyamfi, “Automated road crack detection using deep convolutional neural networks,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 5212–5215.
  * [11] R. Manikandan, S. Kumar, and S. Mohan, “Varying adaptive ensemble of deep detectors for road damage detection,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 5216–5219.
  * [12] V. Pham and T. Dang, “Cvexplorer: Multidimensional visualization for common vulnerabilities and exposures,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 1296–1301.
  * [13] Y. J. Wang, M. Ding, S. Kan, S. Zhang, and C. Lu, “Deep proposal and detection networks for road damage detection and classification,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 5224–5227.
  * [14] W. Wang, B. Wu, S. Yang, and Z. Wang, “Road damage detection and classification with faster r-cnn,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 5220–5223.
  * [15] H. Maeda, T. Kashiyama, Y. Sekimoto, T. Seto, and H. Omata, “Generative adversarial network for road damage detection,” _Computer-Aided Civil and Infrastructure Engineering_ , 2020.
  * [16] D. Arya, H. Maeda, S. K. Ghosh, D. Toshniwal, and Y. Sekimoto, “Rdd2020: An annotated image dataset for automatic road damage detection using deep learning,” _Data in brief_ , vol. 36, p. 107133, 2021.
  * [17] D. Arya, H. Maeda, S. Ghosh, D. Toshniwal, H. Omata, T. Kashiyama, T. Seto, A. Mraz, and Y. Sekimoto, “Rdd2020: an image dataset for smartphone-based road damage detection and classification,” _Mendeley Data_ , vol. 1, 2021\.
  * [18] H. Maeda, Y. Sekimoto, T. Seto, T. Kashiyama, and H. Omata, “Road damage detection and classification using deep neural networks with smartphone images,” _Computer-Aided Civil and Infrastructure Engineering_ , vol. 33, no. 12, pp. 1127–1141, 2018.
  * [19] D. Arya, H. Maeda, S. Kumar Ghosh, D. Toshniwal, H. Omata, T. Kashiyama, and Y. Sekimoto, “Global road damage detection: State-of-the-art solutions,” in _2020 IEEE International Conference on Big Data (Big Data)_ , 2020, pp. 5533–5539.
  * [20] V. Hegde, D. Trivedi, A. Alfarrarjeh, A. Deepak, S. Ho Kim, and C. Shahabi, “Yet another deep learning approach for road damage detection using ensemble learning,” in _2020 IEEE International Conference on Big Data (Big Data)_ , 2020, pp. 5553–5558.
  * [21] K. Doshi and Y. Yilmaz, “Road damage detection using deep ensemble learning,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 5540–5544.
  * [22] Z. Pei, R. Lin, X. Zhang, H. Shen, J. Tang, and Y. Yang, “Cfm: A consistency filtering mechanism for road damage detection,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 5584–5591.
  * [23] V. Mandal, A. R. Mussah, and Y. Adu-Gyamfi, “Deep learning frameworks for pavement distress classification: A comparative analysis,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 5577–5583.
  * [24] D. Jeong, “Road damage detection using yolo with smartphone images,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 5559–5562.
  * [25] Y. Liu, X. Zhang, B. Zhang, and Z. Chen, “Deep network for road damage detection,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 5572–5576.
  * [26] S. Naddaf-Sh, M.-M. Naddaf-Sh, A. R. Kashani, and H. Zargarzadeh, “An efficient and scalable deep learning approach for road damage detection,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 5602–5608.
  * [27] X. Zhang, X. Xia, N. Li, M. Lin, J. Song, and N. Ding, “Exploring the tricks for road damage detection with a one-stage detector,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 5616–5621.
  * [28] T. Hascoet, Y. Zhang, A. Persch, R. Takashima, T. Takiguchi, and Y. Ariki, “Fasterrcnn monitoring of road damages: Competition and deployment,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 5545–5552.
  * [29] A. Behzadian, T. W. Muturi, T. Zhang, H. Kim, A. Mullins, Y. Lu, N. J. Owor, Y. Adu-Gyamfi, W. Buttlar, M. Hamed _et al._ , “The 1st data science for pavements challenge,” _arXiv preprint arXiv:2206.04874_ , 2022.
  * [30] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [31] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” _International journal of computer vision_ , vol. 111, no. 1, pp. 98–136, 2015.
  * [32] D. Arya, H. Maeda, Y. Sekimoto, H. Omata, S. K. Ghosh, D. Toshniwal, M. Sharma, V. V. Pham, J. Zhong, M. Al-Hammadi, M. B. Shami, D. Nguyen, H. Cheng, J. Zhang, A. Klein-Paste, H. Mork, F. Lindseth, T. Seto, A. Mraz, and T. Kashiyama, “RDD2022 - The multi-national Road Damage Dataset released through CRDDC’2022,” 10 2022. [Online]. Available: https://figshare.com/articles/dataset/RDD2022_-_The_multi-national_Road_Damage_Dataset_released_through_CRDDC_2022/21431547
  * [33] D. Arya, H. Maeda, S. K. Ghosh, D. Toshniwal, and Y. Sekimoto, “Rdd2022: A multi-national image dataset for automatic road damage detection,” 2022. [Online]. Available: https://arxiv.org/abs/2209.08538
  * [34] D. Arya, H. Maeda, S. K. Ghosh, D. Toshniwal, A. Mraz, T. Kashiyama, and Y. Sekimoto, “Transfer learning-based road damage detection for multiple countries,” _arXiv preprint arXiv:2008.13101_ , 2020.
  * [35] R. Girshick, “Fast r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1440–1448.
  * [36] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [37] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [38] Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6154–6162.
  * [39] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [40] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271.
  * [41] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [42] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” _arXiv preprint arXiv:2004.10934_ , 2020.
  * [43] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [44] S. Zhang, C. Chi, Y. Yao, Z. Lei, and S. Z. Li, “Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 9759–9768.
  * [45] G. Jocher _et al._ , “ultralytics/yolov5: v6.1 - TensorRT, TensorFlow Edge TPU and OpenVINO Export and Inference,” Feb. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6222936
  * [46] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” _arXiv preprint arXiv:2207.02696_ , 2022.
  * [47] “YOLOv5,” 2020. [Online]. Available: https://github.com/ultralytics/yolov5
  * [48] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * [49] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, F. Wei, and B. Guo, “Swin transformer v2: Scaling up capacity and resolution,” in _International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [51] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 764–773.
  * [52] X. Zhu, H. Hu, S. Lin, and J. Dai, “Deformable convnets v2: More deformable, better results,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 9308–9316.