knowledge graphs, expert systems and their applications in e-commerce and
healthcare.
---|---
|  Zhiqi Shen received the B.Sc. degree in computer science and technology
from Peking University, Beijing, China, the M.Eng. degree in computer
engineering from the Beijing University of Technology, Beijing, and the Ph.D.
degree from Nanyang Technological University, Singapore. He is a Senior
Lecturer and a Senior Research Scientist with the School of Computer Science
and Engineering, Nanyang Technological University. His current research
interests include multiagent systems, goal-oriented modeling, agent augmented
interactive media, and interactive storytelling.
---|---
|  Lizhen Cui (IET Fellow, IEEE Senior Member) is a professor and doctoral
supervisor at School of Software, Shandong University. He is the Co-Director
of Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR) and
Research Center of Software and Data Engineering, Shandong University. He
received his B.Sc., M.Sc., and Ph.D. degrees from Shandong University, Jinan,
China, in 1999, 2002 and 2005, respectively. He has published over 200 high-
level academic articles in TKDE, TPAMI, TPDS, TSC, TCC, TNNLS, AAAI, SIGIR,
CIKM, BIBM, ICDCS, DASFAA, ICWS, ICSOC, AAMAS, and MICCAI. He regularly serves
as PC for prestigious conferences (_i.e.,_ KDD, IJCAI, and CIKM) and reviewers
for IEEE/ACM Transactions Journals. His research interests include big data
analysis, data mining, and crowd science.
---|---

Supplementary Materials

[Empirical Evaluations and Analysis] We reorganize the publicly available
benchmark multivariate time series datasets, and select 9 commonly used
datasets for evaluation. Then, we introduce the experimental setup and
evaluation metrics used to assess the performance. Finally, we present a
comprehensive comparison of the selected state-of-the-art models, with a
particular focus on contrasting methods.

###  -C Datasets

In the field of multivariate time series analysis, there are mainly from 3
archives of datasets, including
UCI222http://archive.ics.uci.edu/ml/datasets.php,
UEA333http://www.timeseriesclassification.com/dataset.php and
MTS444http://www.mustafabaydogan.com/multivariate-time-series-discretization-
for-classification.html. Particularly, it is important to note that there may
be some overlap and discrepancies within these datasets. In this work, we
carefully reorganize the multivariate time series datasets considering
multiple aspects, including application domains, sequence lengths, number of
samples, variables, and classes, to ensure our evaluation cover datasets with
diverse characteristics. Our reorganization allows for a more systematic and
comprehensive analysis of the datasets in relation to their specific
characteristics.

  * •

UCI archive: The UCI archive contains 85 multivariate time series datasets
specifically curated for classification tasks. These datasets encompass a wide
range of application fields, such as audio spectra classification, business,
ECG/EEG classification, human activity recognition, gas detection, motion
classification, etc. More details are listed in Table VIII.

  * •

UEA archive: The UEA archive collects 30 multivariate time series
classification datasets that span various application domains, including audio
spectra classification, ECG/EEG/MEG classification, human activity
recognition, motion classification, etc. More details are listed in Table IX.

  * •

MTS archive: The MTS archive, also known as Baydogan’s archive, consists of 13
multivariate time series datasets covering applications of audio spectra
classification, ECG classification, human activity recognition, motion
classification, etc. More details are listed in Table X.

Additionally, we also prioritize datasets that have been widely adopted in
existing research works. Considering the fact that MTS has lower usage and has
significant overlap with UEA and UCI, we ultimately select 9 representative
datasets primarily from the UCI and UEA archives for evaluation. These
datasets include PhonemeSpectra (PS), DuckDuckGeese (DDG), EigenWorms (EW),
PenDigits (PD), Epileptic Seizure Recognition (Epilepsy), FingerMovements
(FM), Human Activity Recognition Using Smartphones (HAR), Smartphone and
Smartwatch Activity and Biometrics Dataset (WISDM) and UniMiB SHAR (SHAR).
They encompass various application fields, such as digits, images, and
biology. For datasets from UEA, we utilize the pre-defined train-test split.
For the remaining datasets from the UCI archive, we partition the data into
80% for training and 20% for testing. Detailed information about the selected
datasets can be found in Table VII.

Task | Archive | Dataset | $N$ | $V$ | Classes | Length
---|---|---|---|---|---|---
ASC | UEA | PS | 6,668 | 11 | 39 | 217
| UEA | DDG | 100 | 1,345 | 5 | 270
MC | UEA | EW | 259 | 6 | 5 | 17,984
| UCI/UEA | PD | 10,992 | 2 | 10 | 8
EC | UCI | Epilepsy | 12,500 | 1 | 2 | 178
| UEA | FM | 416 | 28 | 2 | 50
HAR | UCI | HAR | 10,299 | 9 | 6 | 128
| UCI | WISDM | 4,091 | 3 | 6 | 200
| Others | SHAR | 11,771 | 3 | 17 | 151
TABLE VII: Statistical information of the selected time series classification datasets for evaluation. ASC refers to Audio Spectra Classification, MC refers to Motion Classification, EC refers to ECG/EEG/EMG/MEG Classification, and HAR refers to Human Activity Recognition. $N$ denotes the number of samples and $V$ denotes the number of variables. Classes refer to the number of classes in each dataset and Length refers to the length of each time series. Dataset | Application | $N$ | $V$ |
---|---|---|---|---
FMA: A Dataset For Music Analysis | Audio Spectra Classification | 106,574 | 518 |
Japanese Vowels | Audio Spectra Classification | 640 | 12 |
Spoken Arabic Digit | Audio Spectra Classification | 8,800 | 13 |
BAUM-1 | Audio Spectra Classification | 1,184 | - |
BAUM-2 | Audio Spectra Classification | 1,047 | - |
3W dataset | Business | 1984 | 8 |
CNNpred: CNN-based stock market prediction using a diverse set of variables | Business | 1,985 | 84 |
Dow Jones Index | Business | 750 | 16 |
ISTANBUL STOCK EXCHANGE | Business | 536 | 8 |
Machine Learning based ZZAlpha Ltd. Stock Recommendations 2012-2014 | Business | 314,080 | - |
Online Retail | Business | 541,909 | 8 |
Online Retail II | Business | 1,067,371 | 8 |
MHEALTH Dataset | ECG Classification | 120 | 23 |
EEG Eye State | EEG Classification | 14,980 | 15 |
EEG Steady-State Visual Evoked Potential Signals | EEG Classification | 9,200 | 16 |
Epileptic Seizure Recognition | EEG Classification | 11,500 | 179 |
EMG data for gestures | EMG Classification | 30,000 | 6 |
EMG Physical Action Data Set | EMG Classification | 10,000 | 8 |
sEMG for Basic Hand movements | EMG Classification | 3,000 | 2,500 |
Robot Execution Failures | Failure Detection | 463 | 90 |
AI4I 2020 Predictive Maintenance Dataset | Failure Detection | 10,000 | 14 |
Breath Metabolomics | Gas Detection | 104 | 1,656 |
Gas Sensor Array Drift Dataset at Different Concentrations | Gas Detection | 13,910 | 129 |
Gas sensor array exposed to turbulent gas mixtures | Gas Detection | 180 | 150,000 |
Gas sensor array temperature modulation | Gas Detection | 4,095,000 | 20 |
Gas sensor array under dynamic gas mixtures | Gas Detection | 4,178,504 | 19 |
Gas sensor array under flow modulation | Gas Detection | 58 | 120,432 |
Gas sensor arrays in open sampling settings | Gas Detection | 18,000 | 1,950,000 |
Gas sensors for home activity monitoring | Gas Detection | 919,438 | 11 |
Ozone Level Detection | Gas Detection | 2,536 | 73 |
Twin gas sensor arrays | Gas Detection | 640 | 480,000 |
Activities of Daily Living (ADLs) Recognition Using Binary Sensors | Human Activity Recognition | 2,747 | - |
Activity Recognition from Single Chest-Mounted Accelerometer | Human Activity Recognition | - | - |
Activity Recognition system based on Multisensor data fusion (AReM) | Human Activity Recognition | 42,240 | 6 |
Bar Crawl: Detecting Heavy Drinking | Human Activity Recognition | 14,057,567 | 3 |
Basketball dataset | Human Activity Recognition | 10,000 | 7 |
Daily and Sports Activities | Human Activity Recognition | 9,120 | 5,625 |
Daphnet Freezing of Gait | Human Activity Recognition | 237 | 9 |
Dataset for ADL Recognition with Wrist-worn Accelerometer | Human Activity Recognition | 3 | 14 |
Heterogeneity Activity Recognition | Human Activity Recognition | 43,930,257 | 16 |
Human Activity Recognition from Continuous Ambient Sensor Data | Human Activity Recognition | 13,956,534 | 37 |
Human Activity Recognition Using Smartphones | Human Activity Recognition | 10,299 | 561 |
Intelligent Media Accelerometer and Gyroscope (IM-AccGyro) Dataset | Human Activity Recognition | 800 | 9 |
Localization Data for Person Activity | Human Activity Recognition | 164,860 | 8 |
MEx | Human Activity Recognition | 6,262 | 710 |
OPPORTUNITY Activity Recognition | Human Activity Recognition | 2,551 | 242 |
PAMAP2 Physical Activity Monitoring | Human Activity Recognition | 3,850,505 | 52 |
REALDISP Activity Recognition Dataset | Human Activity Recognition | 1,419 | 120 |
selfBACK | Human Activity Recognition | 26,136 | 6 |
Simulated Falls and Daily Living Activities Data Set | Human Activity Recognition | 3,060 | 138 |
Smartphone Dataset for Human Activity Recognition in Ambient Assisted Living (AAL) | Human Activity Recognition | 5,744 | 561 |
Smartphone-Based Recognition of Human Activities and Postural Transitions | Human Activity Recognition | 10,929 | 561 |
User Identification From Walking Activity | Human Activity Recognition | - | - |
Vicon Physical Action Data Set | Human Activity Recognition | 3,000 | 27 |
WISDM Smartphone and Smartwatch Activity and Biometrics Dataset | Human Activity Recognition | 15,630,426 | 6 |
BLE RSSI dataset for Indoor localization | Indoor localization | 23,570 | 5 |
BLE RSSI Dataset for Indoor localization and Navigation | Indoor localization | 6,611 | 15 |
Geo-Magnetic field and WLAN dataset for indoor localisation from wristband and smartphone | Indoor localization | 153,540 | 25 |
Hybrid Indoor Positioning Dataset from WiFi RSSI, Bluetooth and magnetometer | Indoor localization | 1,540 | 65 |
UJIIndoorLoc-Mag | Indoor localization | 40,000 | 13 |
Kitsune Network Attack Dataset | Malware/Attack Identification | 27,170,754 | 115 |
URL Reputation | Malware/Attack Identification | 2,396,130 | 3,231,961 |
Detect Malware Types | Malware/Attack Identification | 7,107 | 280 |
Dynamic Features of VirusShare Executables | Malware/Attack Identification | 107,888 | 482 |
BitcoinHeistRansomwareAddressDataset | Malware/Attack Identification | 2,916,697 | 10 |
Character Trajectories | Motion Classification | 2,858 | 3 |
Gesture Phase Segmentation | Motion Classification | 9,900 | 50 |
Indoor User Movement Prediction from RSS data | Motion Classification | 13,197 | 4 |
Pedestrian in Traffic Dataset | Motion Classification | 4,760 | 14 |
WESAD (Wearable Stress and Affect Detection) | Motion Classification | 63,000,000 | 12 |
Australian Sign Language signs | Motion Classification | 6,650 | 15 |
Australian Sign Language signs (High Quality) | Motion Classification | 2,565 | 22 |
PEMS-SF | Others | 440 | 138,672 |
Parking Birmingham | Others | 35,717 | 4 |
Occupancy Detection | Others | 20,560 | 7 |
Synthetic Control Chart Time Series | Others | 600 | 6 |
Crop mapping using fused optical-radar data set | Others | 325,834 | 175 |
Absenteeism at work | Others | 740 | 21 |
Educational Process Mining (EPM): A Learning Analytics Data Set | Others | 230,318 | 13 |
Productivity Prediction of Garment Employees | Others | 1,197 | 15 |
Buzz in social media | Others | 140,000 | 77 |
Condition monitoring of hydraulic systems | Others | 2,205 | 43,680 |
Open University Learning Analytics dataset | Others | - | - |
Data for Software Engineering Teamwork Assessment in Education Setting | Others | 74 | 102 |
Behavior of the urban traffic of the city of Sao Paulo in Brazil | Others | 135 | 18 |
TABLE VIII: The UCI Multivariate Time Series Classification Archive - 85 Datasets. $N$ denotes the number of samples and $V$ denotes the number of variables. Classes refer to the number of classes in each dataset and Length refers to the length of each time series. Dataset |  Application |  $N$ |  $V$ |  Classes |  Length
---|---|---|---|---|---
DuckDuckGeese |  Audio Spectra Classification |  100 |  1,345 |  5 |  270
Heartbeat |  Audio Spectra Classification |  409 |  61 |  2 |  405
InsectWingbeat |  Audio Spectra Classification |  50,000 |  200 |  10 |  30
JapaneseVowels |  Audio Spectra Classification |  640 |  12 |  9 |  29
PhonemeSpectra |  Audio Spectra Classification |  6,668 |  11 |  39 |  217
SpokenArabicDigits |  Audio Spectra Classification |  8,798 |  13 |  10 |  93
AtrialFibrillation |  ECG Classification |  30 |  2 |  3 |  640
StandWalkJump |  ECG Classification |  27 |  4 |  3 |  2,500
FaceDetection |  EEG/MEG Classification |  9,414 |  144 |  2 |  62
FingerMovements |  EEG/MEG Classification |  416 |  28 |  2 |  50
HandMovementDirection |  EEG/MEG Classification |  234 |  10 |  4 |  400
MotorImagery |  EEG/MEG Classification |  378 |  64 |  2 |  3,000
SelfRegulationSCP1 |  EEG/MEG Classification |  561 |  6 |  2 |  896
SelfRegulationSCP2 |  EEG/MEG Classification |  380 |  7 |  2 |  1,152
BasicMotions |  Human Activity Recognition |  80 |  6 |  4 |  100
Cricket |  Human Activity Recognition |  180 |  6 |  12 |  1,197
Epilepsy |  Human Activity Recognition |  275 |  3 |  4 |  206
ERing |  Human Activity Recognition |  300 |  4 |  6 |  65
Handwriting |  Human Activity Recognition |  1,000 |  3 |  26 |  152
Libras |  Human Activity Recognition |  360 |  2 |  15 |  45
NATOPS |  Human Activity Recognition |  360 |  24 |  6 |  51
RacketSports |  Human Activity Recognition |  303 |  6 |  4 |  30
UWaveGestureLibrary |  Human Activity Recognition |  440 |  3 |  8 |  315
ArticularyWordRecognition |  Motion Classification |  575 |  9 |  25 |  144
CharacterTrajectories |  Motion Classification |  2,858 |  3 |  20 |  182
EigenWorms |  Motion Classification |  259 |  6 |  5 |  17,984
PenDigits |  Motion Classification |  10,992 |  2 |  10 |  8
EthanolConcentration |  Others |  524 |  3 |  4 |  1,751
LSST |  Others |  4,925 |  6 |  14 |  36
PEMS-SF |  Others |  440 |  963 |  7 |  144
TABLE IX: The UEA Multivariate Time Series Classification Archive - 30 Datasets. $N$ denotes the number of samples and $V$ denotes the number of variables. Classes refer to the number of classes in each dataset and Length refers to the length of each time series.  Dataset |  Application |  $N$ |  $V$ |  Classes |  Length
---|---|---|---|---|---
ArabicDigits |  Audio Spectra Classification |  8,800 |  13 |  10 |  4 - 93
JapaneseVowels |  Audio Spectra Classification |  640 |  12 |  9 |  7 - 29
ECG |  ECG Classification |  200 |  2 |  2 |  39 - 152
Libras |  Human Activity Recognition |  360 |  2 |  15 |  45
UWave |  Human Activity Recognition |  4,478 |  3 |  8 |  315
AUSLAN |  Motion Classification |  2,565 |  22 |  95 |  45 - 136
CharacterTrajectories |  Motion Classification |  2,858 |  3 |  20 |  109 - 205
CMUsubject16 |  Motion Classification |  58 |  62 |  2 |  127 - 580
KickvsPunch |  Motion Classification |  26 |  62 |  2 |  274 - 841
WalkvsRun |  Motion Classification |  44 |  62 |  2 |  128 - 1,918
NetFlow |  Others |  1,337 |  4 |  2 |  50 - 997
PEMS |  Others |  440 |  963 |  7 |  144
Wafer |  Others |  1,194 |  6 |  2 |  104 - 198
TABLE X: The MTS Multivariate Time Series Classification Archive - 13
Datasets. $N$ denotes the number of samples and $V$ denotes the number of
variables. Classes refer to the number of classes in each dataset and Length
refers to the length of each time series.

###  -D Experimental Setup

We conduct linear evaluation to assess the performance of state-of-the-art
unsupervised learning models, with a particular focus on various contrastive
methods, on 9 representative datasets mentioned in Section -C. All models are
implemented using the PyTorch framework, and the experimental evaluations are
conducted on an NVIDIA GeForce RTX 3090 GPU for efficient computation. To
ensure a fair comparison, we follow the standard linear benchmarking
evaluation scheme employed in [56, 43, 51]. This involves freezing the
representations pre-trained by each unsupervised model and attaching a linear
classifier on top of the frozen representations. The linear classifier is a
logistic regression classifier, which consists of a fully-connected layer
followed by the softmax activation function. We disable gradient computation
on the inputs to the linear classifier. The classification performance is
measured by using the following 3 commonly used metrics: ACC (Accuracy), MF1
(macro-averaged F1 score) and $\kappa$ (Cohen’s Kappa coefficient). All the
models are trained for 200 epochs. The mean and standard deviation of
empirical results obtained from 5 repeated experiments are reported for each
metric.

  * •

ACC: Accuracy is the ratio of the number of correctly classified samples to
the total number of samples, _i.e.,_ ACC
$=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FP}+\text{FN}}$, where
TP, TN, FP, FN are true positive, true negative, false positive and false
negative.

  * •

MF1: Macro-averaged F1 score is calculated as the arithmetic mean of
individual classes’ F1 score, _i.e.,_ MF1
$=\frac{2\times\text{PR}}{\text{P}+\text{R}}$, where Precision P and Recall R
are calculated by
$\text{P}=\frac{\text{TP}}{\text{TP}+\text{FP}},\text{R}=\frac{\text{TP}}{\text{TP}+\text{FN}}$.

  * •

$\kappa$: Cohen’s Kappa coefficient measures the inter-annotator agreement
that expresses the level of agreement between predicted and true labels on
classification, _i.e.,_ $\kappa=\frac{\text{ACC}-p_{e}}{1-p_{e}}$, where
$p_{e}=\frac{[(\text{TP}+\text{FN})*(\text{TP}+\text{FP})+(\text{FP}+\text{TN})*(\text{FN}+\text{TN})]}{N^{2}}$
is the hypothetical probability of chance agreement, and $N$ denotes the total
number of samples.

Category | Model |  | HAR |  |  | WISDM |  |  | Epilepsy |
---|---|---|---|---|---|---|---|---|---|---
ACC | MF1 | $\kappa$ | ACC | MF1 | $\kappa$ | ACC | MF1 | $\kappa$
Deep Clustering | DeepCluster | 69.95$\pm$2.47 | 67.41$\pm$2.19 | 63.43$\pm$2.20 | 72.89$\pm$2.38 | 70.39$\pm$3.16 | 68.07$\pm$3.45 | 86.07$\pm$1.84 | 80.99$\pm$1.97 | 76.43$\pm$2.18
IDFD | 67.07$\pm$1.62 | 64.12$\pm$1.45 | 60.87$\pm$1.40 | 68.14$\pm$1.82 | 64.78$\pm$1.91 | 61.08$\pm$1.93 | 79.56$\pm$1.73 | 74.84$\pm$1.52 | 70.24$\pm$1.56
Reconstruction-based | TimeNet | 74.61$\pm$2.42 | 73.64$\pm$2.51 | 70.35$\pm$2.73 | 79.23$\pm$2.76 | 71.82$\pm$2.73 | 70.49$\pm$3.04 | 89.32$\pm$1.39 | 85.71$\pm$1.43 | 75.76$\pm$1.92
Deconv | 71.53$\pm$1.65 | 70.76$\pm$1.84 | 62.72$\pm$2.39 | 75.76$\pm$2.30 | 69.45$\pm$2.79 | 66.28$\pm$3.21 | 85.44$\pm$1.92 | 79.85$\pm$2.08 | 76.01$\pm$2.57
Self-supervised | TimeGAN | 73.57$\pm$1.96 | 70.04$\pm$1.94 | 66.71$\pm$2.30 | 79.22$\pm$2.61 | 72.43$\pm$2.88 | 69.26$\pm$3.03 | 89.79$\pm$1.24 | 86.55$\pm$1.36 | 74.83$\pm$1.77
-Adversarial | TimeVAE | 70.69$\pm$2.07 | 65.61$\pm$2.41 | 61.45$\pm$2.89 | 74.46$\pm$2.97 | 67.93$\pm$3.41 | 64.10$\pm$3.94 | 86.91$\pm$1.17 | 82.40$\pm$1.41 | 76.98$\pm$2.56
-Predictive | EEG-SSL | 65.34$\pm$1.63 | 63.75$\pm$1.37 | 57.20$\pm$1.22 | 70.67$\pm$1.11 | 66.50$\pm$0.93 | 62.44$\pm$1.07 | 93.72$\pm$0.45 | 89.15$\pm$0.93 | 77.65$\pm$1.64
| TST | 70.73$\pm$2.29 | 66.36$\pm$2.03 | 63.14$\pm$2.74 | 76.68$\pm$2.85 | 70.06$\pm$2.94 | 67.30$\pm$3.41 | 91.21$\pm$0.88 | 87.64$\pm$1.33 | 87.49$\pm$1.75
-Contrastive | SimCLR | 81.06$\pm$2.35 | 80.62$\pm$2.31 | 77.25$\pm$2.82 | 83.04$\pm$4.21 | 75.83$\pm$6.47 | 75.15$\pm$6.27 | 93.00$\pm$0.57 | 88.09$\pm$0.97 | 76.27$\pm$1.93
| BYOL | 89.46$\pm$0.17 | 89.31$\pm$0.17 | 87.33$\pm$0.20 | 87.84$\pm$0.38 | 84.02$\pm$0.77 | 82.43$\pm$0.57 | 98.08$\pm$0.09 | 96.99$\pm$0.15 | 93.99$\pm$0.30
| SwAV | 68.81$\pm$1.50 | 66.69$\pm$1.56 | 62.41$\pm$1.81 | 73.44$\pm$1.28 | 53.90$\pm$3.56 | 59.75$\pm$2.29 | 94.30$\pm$0.85 | 90.80$\pm$1.37 | 81.62$\pm$2.73
| PCL | 74.49$\pm$1.95 | 65.02$\pm$1.69 | 71.96$\pm$2.12 | 69.47$\pm$1.51 | 61.75$\pm$3.29 | 56.36$\pm$2.57 | 89.93$\pm$1.34 | 87.68$\pm$1.42 | 85.78$\pm$1.88
| TS-TCC | 89.22$\pm$0.70 | 89.23$\pm$0.76 | 87.03$\pm$0.85 | 81.48$\pm$0.98 | 69.17$\pm$2.25 | 73.13$\pm$1.33 | 97.19$\pm$0.18 | 95.47$\pm$0.31 | 90.94$\pm$0.61
| T-Loss | 91.06$\pm$0.94 | 90.94$\pm$0.96 | 89.26$\pm$1.13 | 91.48$\pm$1.05 | 88.79$\pm$1.53 | 87.79$\pm$1.51 | 96.94$\pm$0.20 | 95.20$\pm$0.30 | 90.41$\pm$0.61
TABLE XI: Comparisons between different categories of unsupervised learning
methods. ACC refers to accuracy, MF1 refers to macro-averaged F1 score, and
$\kappa$ refers to Cohen’s kappa coefficient.

###  -E Results and Discussions

We choose 3 time series datasets for conducting comparisons among various
categories of unsupervised learning methods. Following this analysis, due to
the promising results and growing popularity of contrastive learning methods
over other unsupervised learning methods, we allocate special emphasis on
contrastive learning methods covering multiple levels of contrast in the
following subsections.

####  -E1 A comparative analysis of different categories of unsupervised
learning methods

The comparisons between different categories of unsupervised learning methods
on 3 selected datasets are presented in Table XI. In general, self-supervised
learning approaches outperform deep clustering and reconstruction-based
methods. Such performance gain indicates that self-supervised models offer
improved representation learning capabilities. This can be attributed to the
broader range of pretext tasks, such as predicting missing elements, temporal
order, or context, which encourage the model to learn rich and meaningful
representations. This diversity of tasks enables the model to capture
different aspects of the data, leading to more comprehensive and robust
representations. Particularly, contrastive methods have demonstrated promising
results in most datasets as they distinguish samples from multiple views
without the need for explicit labels. Such discrimination allows the learned
representations to be more effective in capturing meaningful differences
between data samples. Instead, adversarial methods such as TimeGAN [69] and
TimeVAE [70] assume specific data distributions, require iterative
optimization and often are more challenging to converge. These methods can be
computationally expensive and require larger amounts of data for training.
Whereas, contrastive methods are less sensitive to the data distribution
compared to adversarial methods and are generally easier to train.
Additionally, predictive methods including EEG-SSL [71] and TST [104], tend to
perform worse than most contrastive methods. This is because predictive
methods rely on a large number of given contexts to accurately predict part of
the data, which can be challenging in practice. Conversely, contrastive
methods can make better use of the available data by focusing on
differentiating between a smaller number of similar and dissimilar data pairs.

Dataset | Level | Model | ACC | MF1 | $\kappa$
---|---|---|---|---|---
Epilepsy | Instance | SimCLR | 93.00$\pm$0.57 | 88.09$\pm$0.97 | 76.27$\pm$1.93
BYOL | 98.08$\pm$0.09 | 96.99$\pm$0.15 | 93.99$\pm$0.30
CPC | 96.61$\pm$0.43 | 94.44$\pm$0.69 | 88.67$\pm$1.22
Prototype | SwAV | 94.30$\pm$0.85 | 90.80$\pm$1.37 | 81.62$\pm$2.73
PCL | 89.93$\pm$1.34 | 87.68$\pm$1.42 | 85.78$\pm$1.88
MHCCL | 97.85$\pm$0.49 | 95.44$\pm$0.82 | 91.08$\pm$1.57
Temporal | TS2Vec | 96.32$\pm$0.23 | 94.27$\pm$0.37 | 88.54$\pm$0.74
TS-TCC | 97.19$\pm$0.18 | 95.47$\pm$0.31 | 90.94$\pm$0.61
T-Loss | 96.94$\pm$0.20 | 95.20$\pm$0.30 | 90.41$\pm$0.61
HAR | Instance | SimCLR | 81.06$\pm$2.35 | 80.62$\pm$2.31 | 77.25$\pm$2.82
BYOL | 89.46$\pm$0.17 | 89.31$\pm$0.17 | 87.33$\pm$0.20
CPC | 83.85$\pm$1.51 | 83.27$\pm$1.66 | 79.76$\pm$1.90
Prototype | SwAV | 68.81$\pm$1.50 | 66.69$\pm$1.56 | 62.41$\pm$1.81
PCL | 74.49$\pm$1.95 | 65.02$\pm$1.69 | 71.96$\pm$2.12
MHCCL | 91.60$\pm$1.06 | 91.77$\pm$1.11 | 89.90$\pm$1.27
Temporal | TS2Vec | 90.47$\pm$0.66 | 90.46$\pm$0.64 | 89.15$\pm$0.79
TS-TCC | 89.22$\pm$0.70 | 89.23$\pm$0.76 | 87.03$\pm$0.85
T-Loss | 91.06$\pm$0.94 | 90.94$\pm$0.96 | 89.26$\pm$1.13
SHAR | Instance | SimCLR | 67.22$\pm$1.76 | 53.05$\pm$2.42 | 63.84$\pm$1.97
BYOL | 67.00$\pm$0.98 | 59.30$\pm$1.30 | 63.62$\pm$1.10
CPC | 68.57$\pm$1.40 | 62.44$\pm$1.77 | 61.62$\pm$0.96
Prototype | SwAV | 57.49$\pm$2.75 | 58.82$\pm$2.53 | 58.89$\pm$3.50
PCL | 56.28$\pm$1.47 | 49.42$\pm$1.62 | 51.78$\pm$1.65
MHCCL | 83.42$\pm$1.76 | 78.45$\pm$2.09 | 80.58$\pm$1.84
Temporal | TS2Vec | 82.94$\pm$2.91 | 77.89$\pm$2.95 | 78.94$\pm$3.20
TS-TCC | 70.80$\pm$0.75 | 64.28$\pm$0.72 | 67.74$\pm$0.85
T-Loss | 80.88$\pm$3.94 | 77.06$\pm$3.44 | 79.65$\pm$4.35
WISDM | Instance | SimCLR | 83.04$\pm$4.21 | 75.83$\pm$6.47 | 75.15$\pm$6.27
BYOL | 87.84$\pm$0.38 | 84.02$\pm$0.77 | 82.43$\pm$0.57
CPC | 80.35$\pm$0.98 | 73.24$\pm$1.21 | 72.23$\pm$1.07
Prototype | SwAV | 73.44$\pm$1.28 | 53.90$\pm$3.56 | 59.75$\pm$2.29
PCL | 69.47$\pm$1.51 | 61.75$\pm$3.29 | 56.36$\pm$2.57
MHCCL | 93.60$\pm$1.06 | 91.70$\pm$1.10 | 90.96$\pm$1.08
Temporal | TS2Vec | 92.33$\pm$1.05 | 90.27$\pm$1.07 | 90.36$\pm$1.49
TS-TCC | 81.48$\pm$0.98 | 69.17$\pm$2.25 | 73.13$\pm$1.33
T-Loss | 91.48$\pm$1.05 | 88.79$\pm$1.53 | 87.79$\pm$1.51
TABLE XII: A comparative analysis of different levels of contrast on 4 datasets. ACC refers to accuracy, MF1 refers to macro-averaged F1 score, and $\kappa$ refers to Cohen’s kappa coefficient. Dataset | Level | Model | ACC | MF1 | $\kappa$
---|---|---|---|---|---
| Instance | SimCLR | 40.00$\pm$0.60 | 39.01$\pm$0.69 | 25.00$\pm$0.74
BYOL | 51.60$\pm$1.20 | 52.04$\pm$1.34 | 39.50$\pm$1.50
CPC | 44.72$\pm$2.19 | 42.90$\pm$3.45 | 30.77$\pm$4.16
Prototype | SwAV | 52.40$\pm$3.44 | 50.95$\pm$4.46 | 40.50$\pm$4.30
DDG | PCL | 48.62$\pm$3.56 | 46.86$\pm$3.97 | 35.67$\pm$5.74
| MHCCL | 50.76$\pm$3.19 | 49.52$\pm$3.54 | 39.50$\pm$4.27
| Temporal | TS2Vec | 52.00$\pm$3.27 | 51.02$\pm$3.64 | 40.00$\pm$4.08
| TS-TCC | 38.67$\pm$5.96 | 35.51$\pm$6.48 | 23.33$\pm$7.45
| T-Loss | 54.50$\pm$4.33 | 52.70$\pm$5.48 | 43.13$\pm$5.41
| Instance | SimCLR | 49.20$\pm$0.98 | 43.08$\pm$4.90 | -0.69$\pm$2.29
BYOL | 49.60$\pm$1.11 | 49.38$\pm$1.12 | -0.49$\pm$2.22
CPC | 49.73$\pm$1.45 | 44.76$\pm$1.69 | -0.27$\pm$2.70
Prototype | SwAV | 44.22$\pm$2.19 | 44.19$\pm$2.20 | 8.47$\pm$4.40
FM | PCL | 50.44$\pm$1.95 | 43.68$\pm$1.30 | 1.19$\pm$3.89
| MHCCL | 52.09$\pm$1.45 | 50.51$\pm$2.06 | 17.87$\pm$3.39
| Temporal | TS2Vec | 50.00$\pm$1.63 | 49.99$\pm$1.64 | -0.01$\pm$3.30
| TS-TCC | 50.17$\pm$1.60 | 49.07$\pm$2.10 | 0.33$\pm$4.58
| T-Loss | 50.50$\pm$1.72 | 50.33$\pm$1.76 | 4.01$\pm$6.65
| Instance | SimCLR | 93.35$\pm$0.17 | 93.27$\pm$0.17 | 92.61$\pm$0.19
BYOL | 94.93$\pm$0.08 | 94.96$\pm$0.07 | 94.37$\pm$0.09
CPC | 92.03$\pm$0.32 | 91.00$\pm$0.35 | 90.04$\pm$0.36
Prototype | SwAV | 78.02$\pm$2.86 | 76.86$\pm$3.06 | 71.10$\pm$3.19
PD | PCL | 86.18$\pm$1.25 | 83.36$\pm$1.11 | 80.69$\pm$1.01
| MHCCL | 98.69$\pm$0.41 | 98.71$\pm$0.55 | 97.43$\pm$0.72
| Temporal | TS2Vec | 97.83$\pm$0.24 | 97.80$\pm$0.25 | 97.59$\pm$0.27
| TS-TCC | 97.44$\pm$0.23 | 97.45$\pm$0.23 | 97.16$\pm$0.26
| T-Loss | 97.86$\pm$0.52 | 97.87$\pm$0.52 | 97.63$\pm$0.57
| Instance | SimCLR | 15.14$\pm$0.30 | 12.64$\pm$0.33 | 12.91$\pm$0.31
BYOL | 12.08$\pm$0.21 | 11.62$\pm$0.19 | 9.77$\pm$0.21
CPC | 14.31$\pm$0.78 | 14.32$\pm$0.77 | 9.95$\pm$0.84
Prototype | SwAV | 11.66$\pm$0.64 | 8.92$\pm$0.58 | 9.33$\pm$0.65
PS | PCL | 8.92$\pm$1.04 | 8.96$\pm$1.24 | 6.53$\pm$1.49
| MHCCL | 15.43$\pm$0.79 | 14.29$\pm$0.76 | 12.63$\pm$0.79
| Temporal | TS2Vec | 15.69$\pm$0.90 | 15.69$\pm$0.95 | 13.47$\pm$0.92
| TS-TCC | 10.90$\pm$0.76 | 9.16$\pm$0.65 | 8.56$\pm$0.78
| T-Loss | 18.60$\pm$0.25 | 18.15$\pm$0.28 | 16.46$\pm$0.25
| Instance | SimCLR | 60.87$\pm$1.84 | 55.62$\pm$4.69 | 47.99$\pm$3.27
BYOL | 79.39$\pm$0.84 | 75.12$\pm$1.05 | 71.84$\pm$1.19
CPC | 67.03$\pm$1.38 | 63.06$\pm$1.38 | 58.26$\pm$1.53
Prototype | SwAV | 43.75$\pm$1.80 | 35.16$\pm$2.31 | 21.82$\pm$2.27
EW | PCL | 43.28$\pm$1.06 | 40.02$\pm$2.01 | 22.58$\pm$1.59
| MHCCL | 79.10$\pm$2.21 | 76.01$\pm$1.45 | 69.53$\pm$1.90
| Temporal | TS2Vec | 82.80$\pm$2.52 | 81.59$\pm$3.73 | 80.74$\pm$3.41
| TS-TCC | 73.21$\pm$1.44 | 64.33$\pm$0.81 | 63.41$\pm$1.17
| T-Loss | 75.00$\pm$3.39 | 67.85$\pm$3.64 | 65.12$\pm$4.91
TABLE XIII: A comparative analysis of different levels of contrast on UEA
datasets. ACC refers to accuracy, MF1 refers to macro-averaged F1 score, and
$\kappa$ refers to Cohen’s kappa coefficient.

####  -E2 A comparative analysis of performing different levels of contrast

We conducted a specific comparison and analysis of contrastive methods
focusing on different levels of contrast. Table XIII shows the comparative
analysis at different levels of contrast on UEA datasets, and the results on 4
other datasets are shown in Table XII. From the results, it can be observed
that contrastive learning models such as TS2Vec [52], TS-TCC [51] and T-Loss
[50], which emphasize the impact at the temporal level, achieve better results
compared to methods that focus on instance-level or prototype-level contrast.
However, prototype-level contrastive learning models such as SwAV [59] and PCL
[58], which have shown great success in image data, do not perform as well in
time series data. These methods usually employ flat clustering algorithms,
which are capable of capturing only a single hierarchy of semantic clusters.
Nonetheless, the result obtained from one clustering iteration cannot be
consistently relied upon as there is no assurance of their accuracy, rendering
them inadequate. In this case, it is worth noting that MHCCL [60] demonstrates
superior performance as it incorporates hierarchical clustering into the
construction of contrastive pairs, which helps address the limitation of flat
clustering algorithms.

####  -E3 A comparative analysis of using different backbones

Table XIV presents a comparative analysis of employing different backbones in
the SimCLR model for unsupervised time series representation learning.
Typically, we find that most models summarized in Table IV, Table V and Table
VI utilize ResNets [145] as the backbone and achieve satisfactory performance.
Therefore, we compare the results of using ResNet-18 and ResNet-50 as
backbones. Furthermore, we evaluate InceptionTime [146], which is specifically
designed for time series data by incorporating the temporal dependencies. As
shown in Table XIV, InceptionTime outperforms the general-purpose ResNets in
most datasets. This observation highlights the capability of InceptionTime to
identify both local and global shape patterns (_i.e.,_ low-level and high-
level features) in time series data. InceptionTime proves to be exceptionally
fitting for time series data due to its utilization of 1D convolutions with
multiple kernel sizes in parallel. This approach facilitates efficient feature
extraction on time series data across various scales, making it well-suited to
capture diverse granularities of semantics in time series data. In contrast,
ResNets primarily rely on 2D convolutions and are optimized for image data,
which may not yield the same level of effectiveness when applied to time
series data analysis.

Dataset | Metric | ResNet-18 | ResNet-50 | InceptionTime
---|---|---|---|---
HAR | ACC | 78.70$\pm$2.27 | 81.06$\pm$2.35 | 82.25$\pm$1.94
MF1 | 78.44$\pm$2.19 | 80.62$\pm$2.31 | 80.56$\pm$2.28
$\kappa$ | 76.21$\pm$2.43 | 77.25$\pm$2.82 | 78.39$\pm$2.65
WISDM | ACC | 86.94$\pm$1.92 | 83.04$\pm$4.21 | 87.46$\pm$1.87
MF1 | 81.65$\pm$3.26 | 75.83$\pm$6.47 | 82.14$\pm$2.85
$\kappa$ | 80.91$\pm$2.95 | 75.15$\pm$6.27 | 81.62$\pm$2.24
Epilepsy | ACC | 93.42$\pm$0.26 | 93.00$\pm$0.57 | 93.68$\pm$0.41
MF1 | 88.72$\pm$0.54 | 88.09$\pm$0.97 | 88.80$\pm$0.64
$\kappa$ | 77.54$\pm$1.06 | 76.27$\pm$1.93 | 77.89$\pm$1.39
TABLE XIV: The classification performance of using different backbones in
SimCLR. ACC refers to accuracy, MF1 refers to macro-averaged F1 score, and
$\kappa$ refers to cohen’s kappa coefficient.

####  -E4 A comparative analysis of employing different augmentation selection
strategies

Table XV presents the classification accuracy of employing different data
augmentation selection strategies. We evaluate 4 strategies in 3 popular
contrastive learning models on 3 datasets. As shown in Table XV, we compare
the original strategy (Original) used in each model with 3 alternative
augmentation selection strategies. The specified one (Specified) follows the
implementation in TS-TCC [51] which employs simple yet efficient
transformations that can fit any time series data to create a strong and a
weak view. Therefore, the specified strategy is equivalent to the original one
in TS-TCC model. Both the aforementioned original and specified augmentation
selection strategies are developed based on domain expert knowledge and
empirical results. Additionally, we also compare with 2 automatic selection
strategies including RandAugment [147] and W-Augment [148]. Nevertheless, it
is noteworthy that the advancements achieved through the utilization of
automatic augmentation selection strategies have shown minimal improvement. In
some cases, the performance does not surpass that of the original performance.
Consequently, developing automatic augmentation selection strategies
specifically tailored for time series data or optimizing the combinations of
various transformations remains an open problem for boosting contrastive
learning.

Dataset | Strategy | SimCLR | TS-TCC | BYOL
---|---|---|---|---
HAR | Original | 81.83$\pm$2.18 | 89.22$\pm$0.70 | 89.36$\pm$2.09
Specified | 81.06$\pm$2.35 | 89.22$\pm$0.70 | 89.46$\pm$0.17
RandAugment | 81.02$\pm$2.37 | 87.94$\pm$2.14 | 89.90$\pm$2.37
W-Augment | 81.27$\pm$1.66 | 87.57$\pm$1.69 | 89.35$\pm$1.96
WISDM | Original | 83.71$\pm$3.48 | 81.48$\pm$0.98 | 87.91$\pm$2.42
Specified | 83.04$\pm$4.21 | 81.48$\pm$0.98 | 87.84$\pm$0.38
RandAugment | 82.78$\pm$2.69 | 82.32$\pm$2.36 | 88.03$\pm$2.99
W-Augment | 84.30$\pm$1.85 | 82.47$\pm$1.87 | 87.84$\pm$2.25
Epilepsy | Original | 92.97$\pm$1.62 | 97.19$\pm$0.18 | 97.64$\pm$1.23
Specified | 93.00$\pm$0.57 | 97.19$\pm$0.18 | 98.08$\pm$0.09
RandAugment | 93.26$\pm$2.17 | 98.04$\pm$0.64 | 98.06$\pm$0.91
W-Augment | 92.19$\pm$1.79 | 97.52$\pm$0.92 | 97.67$\pm$0.79
TABLE XV: The classification accuracy of employing different data augmentation
selection strategies.


11institutetext: Dpt. of Computing and Informatics, Bournemouth University,
UK, 11email<EMAIL_ADDRESS>22institutetext: Australian Institute
for Machine Learning, University of Adelaide, Australia 33institutetext: BCN
Medtech, Dept. of Information and Communication Technologies,
Universitat Pompeu Fabra, Barcelona, Spain 44institutetext: ICREA, Barcelona,
Spain

# Double Encoder-Decoder Networks
for Gastrointestinal Polyp Segmentation

Adrian Galdran 11 Gustavo Carneiro 22 Miguel A. González Ballester 3344

###### Abstract

Polyps represent an early sign of the development of Colorectal Cancer. The
standard procedure for their detection consists of colonoscopic examination of
the gastrointestinal tract. However, the wide range of polyp shapes and visual
appearances, as well as the reduced quality of this image modality, turn their
automatic identification and segmentation with computational tools into a
challenging computer vision task. In this work, we present a new strategy for
the delineation of gastrointestinal polyps from endoscopic images based on a
direct extension of common encoder-decoder networks for semantic segmentation.
In our approach, two pretrained encoder-decoder networks are sequentially
stacked: the second network takes as input the concatenation of the original
frame and the initial prediction generated by the first network, which acts as
an attention mechanism enabling the second network to focus on interesting
areas within the image, thereby improving the quality of its predictions.
Quantitative evaluation carried out on several polyp segmentation databases
shows that double encoder-decoder networks clearly outperform their single
encoder-decoder counterparts in all cases. In addition, our best double
encoder-decoder combination attains excellent segmentation accuracy and
reaches state-of-the-art performance results in all the considered datasets,
with a remarkable boost of accuracy on images extracted from datasets not used
for training.

###### Keywords:

Polyp Segmentation Colonoscopy Colorectal Cancer

(a)

(b)

(c)

(d)

Figure 1: Polyp visual aspects have a wide variety in terms of shape and
color. Four different polyps sampled from the different databases considered
in this work: (a) Kvasir-Seg [19], (b) CVC-ClinicDB [5], (c) CVC-ColonDB [4],
(d) ETIS [29].

##  1 Introduction

The large bowel within the human gastrointestinal tract can be affected by
different diseases, among which, Colorectal Cancer (CRC) is particularly
concerning. CRC represents the second most common cancer type in women and
third most common for men [15]. Gastro-intestinal polyps are known precursors
of this type of cancer [34], being present in almost half of the patients over
50 undergoing screening colonoscopies [31]. This kind of lesions show a wide
range of shapes and visual appearances, as shown in Fig. 1, turning its
identification and segmentation into a challenging problem.

Treatment of CRC starts with the detection of colorectal paraneoplastic
lesions, performed during colonoscopy screenings. In this endoscopic
procedure, a flexible tube with a light camera mounted on its tip is
introduced through the rectum to find and sample (or resect) polyps from the
colon. Early detection of CRC has been demonstrated to substantially increase
5-year survival rates, with screening programs enabling even pre-symptomatic
treatment [31]. Unfortunately, it is estimated that around 6-27% of polyps are
missed during a colonoscopic examination [1]. It has been recently shown in
[25] that up to 80% of missed lesions could be avoided with real-time
computer-aided colonoscopic image analysis and decision support systems.
Therefore, computer-aided polyp detection has been extensively explored as a
complementary tool for colonoscopic procedures to improve detection rates,
enable early treatment, and increase survival rates.

The most relevant computer-aided tasks related to polyp analysis in endoscopic
imaging are:  1. Polyp Detection: Deciding if polyps appear in an endoscopic
frame [6].  2. Polyp Classification: Assigning polyps to a range of sub-
categories or degrees of malignancy [7].  3. Polyp Localization: finding the
position (usually in terms of a bounding box) of polyps within a frame [38].
4. Polyp Segmentation: delineating the exact polyp contour in a given
endoscopic frame [35].

This work is concerned with the task of polyp segmentation, which has
attracted much attention in recent years. Polyp segmentation is typically
achieved by means of encoder-decoder architectures composed a pair of
Convolutional Neural Networks (CNN). The main variation across different works
is the design of such architectures. For instance in [14] an encoder-decoder
network containing multi-resolution, multi-classification, and fusion sub-
networks was proposed, and in [32] several combinations of different encoder
and decoder architectures were explored. In [12] an architecture in which
there is a shared encoder and two mutually depending decoders that model polyp
areas and boundaries respectively is introduced, whereas in [22] ensembles of
instance-segmentation architectures like Mask-RCNN were studied, and in [11]
parallel reverse attention layers are proposed to better capture the
relationship between polyp areas and their boundaries. Two-stage
detection/segmentation pipelines, in which a first an object detector roughly
locates a polyp and then an encoder-decoder CNN delineates its margins have
also been explored, e.g. in [21]. Alternative loss function have also been
studied, e.g. in [32]. A recent review of these and other approaches can be
found in [31].

The approach proposed here consists of stacking two segmentation networks in a
sequential manner, where the second network receives as input the
concatenation of the prediction from the first one with the original frame, as
shown in Fig. 2. This way, the output of the first network acts as an
attention mechanism that provides the second network with a map of interesting
locations on which the second network should focus. It should be noted that
iterative segmentation networks have been proposed in other contexts recently,
such as the method by Li et al. [23], which proposes stacking a large U-Net
with $N-1$ smaller U-Nets for the task of retinal vessel segmentation. The
main difference compared with our technique is that in [23] the subsequent
small U-Nets do not process the input image, but rather the candidate
segmentation only. Also for retinal vessel segmentation, the recent paper by
Galdran et al. [13] makes use of two consecutive U-Nets, but their focus is on
building compact models for a substantially simpler task, not benefiting from
using pre-trained CNN. The Double-UNet, introduced in [18], holds also some
similarities with our proposed approach, but in their approach the output of
the first network is multiplied by the input image (instead of concatenated),
potentially discarding useful information, which may not be recovered, for the
second network. In addition, the backbone encoder for Double-Unet is a VGG-19
[30], which is currently not competitive with the state-of-the-art
classification networks that have been proposed in recent years.

Figure 2: Pre-trained Double Encoder-Decoder Network for Polyp Segmentation.
The second network receives as input the original colonoscopic frame
concatenated with the prediction of the first network; we conjecture that this
allows the second network to better focus on interesting areas of the image,
improving its segmentation accuracy.

In this work, we propose a framework for semantic segmentation based on the
sequential use of two encoder-decoder networks. Extensive experimental results
with three different pre-trained encoders and decoders on a recently proposed
polyp segmentation dataset [19] show that our double encoder-decoder networks
provide a boost in performance over their single encoder-decoder in all cases.
In addition, results across different combinations of encoders show
substantial robustness and consistency. Our best model is re-trained on a
common polyp segmentation benchmark in order to enable comparison with the
state-of-the-art, attaining results that considerably improve upon several
recent approaches, with remarkable performance on three unseen datasets, where
our approach achieves a performance boost in terms of Dice score of 1.52%,
5,29%, and 12,07% when compared to the second best considered method.

##  2 Methodology

###  2.1 Double Autoencoders

Dense semantic segmentation tasks that produce per-pixel predictions are
typically approached with an encoder-decoder network [3, 33, 36]. In this
context, the encoder can be regarded as a feature extractor that downsamples
spatial resolution and increases the number of channels by learning
convolutional filters. After the image has been encoded into a low-dimensional
representation, the decoder upsamples this representation back to the original
input size, thereby generating a pixel-wise prediction. Typically, skip
connections are added to map information from the encoder to the decoder [27].
Double encoder-encoders are a direct extension of encoder-decoder
architectures in which two encoder-decoder networks are sequentially combined.
Denoting by $x$ the input RGB image, $E^{(1)}$ the first network, and
$E^{(2)}$ the second network, in a double encoder-decoder, the output
$E^{(1)}(x)$ of the first network is provided to the second network together
with $x$ so that it can act as an attention map that allows $E^{(2)}$ to focus
on the most interesting areas of the image:

$E(x)=E^{(2)}(x,E^{(1)}(x)),$ (1)

where $x$ and $E^{(1)}(x)$ are stacked so that the input to $E^{(2)}$ has four
channels instead of the three channels corresponding to the RGB components of
$x$. This is illustrated in Fig. 2. There are some choices to be made in this
framework, specifically about the structure of the encoder and decoder sub-
networks within $E^{(1)}$ and $E^{(2)}$. Note that $E^{(1)}$ and $E^{(2)}$ do
not need to share the same architecture, although in this work we restrict
ourselves to this case to simplify the exposition. In the next two sections we
describe different alternatives that we will thoroughly explore in our
experiments.

###  2.2 Pretrained Encoders

Double encoder-decoder networks contain two sub-networks $E^{(1)}$ and
$E^{(2)}$, each of which have an encoder branch that can be chosen as any
Convolutional Neural Network designed for classification purposes. Image
representations obtained at the end of this branch are expected to be useful
to classify objects within the image and to segment objects when iteratively
upscaled in the decoder branch. In particular, it is favorable to select top-
performer architectures that have been shown to work well for natural image
classification tasks, since we can then re-use the weights of the decoder.
This experimentally leads to faster convergence, requires less training data
and leads to better generalization. In this work we consider three such
architectures, which we briefly review below. These range from more compact
models to larger, more powerful architectures, to better illustrate the
benefits of double encoder-decoder networks under different scenarios.

####  2.2.1 MobileNet V2

MobileNet is a mobile architecture specifically designed to maximize
compactness, speed, and efficiency [28, 28] to enable its deployment on mobile
devices. The main feature of this network is that it implements efficient
depthwise separable convolutions to reduce computational load. Later improved
versions of this architecture adopt “inverted residual” blocks on which
shortcut connections are established between “thin” activation volumes that
contain less features, considerably reducing memory constraints. In addition,
interleaved non-linearities within narrow layers of the network are avoided,
which is shown to further decrease computational demands.

####  2.2.2 Resnet34

Residual Network architectures (ResNets) were first introduced in [16], and
have since then become the default networks for Computer Vision tasks, due to
their excellent complexity versus performance trade-off. ResNets popularized
the notion of skip-connections, which were the main ingredient of residual
blocks: a series of convolutions are applied to the input of a particular
network layer, but a by-pass is added to the input so that information can
flow unaltered up to the output of the block. This improves the training
process of very deep networks, helping to avoid vanishing gradient problems.
Although large ResNet architectures with up to 152 layers have been proposed,
as well as improved versions like ResNext [37], we limit our experiments to a
relatively small ResNet variant, namely ResNet34. This provides a nice
intermediate stage in between the compactness of MobileNet and the complexity
of Dual-Path-Networks, which we describe next.

####  2.2.3 Dual-Path Networks

Dual Path Networks (DPNs) were presented in [10], and they provide an hybrid
design that merges different aspects of Residual Networks and Densely
Connected CNNs (DenseNets, [17]). As described above, ResNets employ skip-
connections within each single residual block to enable better information
flow. DenseNets in turn introduce these paths between all residual blocks,
concatenating input and output features from different network levels. The
authors analyze both architectures and discover that residuals path implicitly
favor features re-usage, but are limited in exploring new features, whereas
DenseNets are better at exploring new features but suffer from heavy feature
redundancy. This observation leads to the proposal of a combined architecture
that is shown to outperform both ResNets and DenseNets in a variety of
computer vision tasks.

###  2.3 Decoders

It is common in semantic segmentation architectures to decouple the design of
the encoder and the decoder, as this provides a better understanding of each
component [36]. In addition, it is usual to adopt pretrained encoders as
described above, which leaves some design freedom to focus on specific
characteristics of encoders that may allow for better reconstruction and
upsampling techniques. We review below the three encoder architectures that we
will be using in our experimental evaluation.

####  2.3.1 U-Net decoder

We first adopt the encoder branch of the widely popular U-Net architecture,
one of the first CNNs designed for image segmentation, with particular success
in biomedical imaging [27]. U-Net was among the first architecture designs to
introduce the idea of an encoding (downsampling) branch followed by a decoding
(upsampling) branch, with skip connections mapping intermediate activation
from one branch to the other. This was shown to improve gradient flow during
backpropagation and iterative weight updating, being also useful for
recovering high-resolution information from the downsampling branch and adding
it to the upsampling activations. The decoder branch of the U-Net features
transposed convolutions as the most distinctive characteristic.

####  2.3.2 DeepLab

The first version of this architecture was presented in [8], with the main
feature being that atrous convolution that were leveraged to gain control on
the resolution at which feature responses are computed by the network. Further
improvements have been introduced to this architecture later on, and in this
work we adopt DeepLab V3 [9], which differs form DeepLab V1 on that atrous
spatial pyramid pooling and batch-norm layers are adopted to increase long-
range information modeling and improve training efficiency.

####  2.3.3 Feature-Pyramid Networks

FPNs were introduced in [24]. The authors proposed to build segmentation
predictions not only in the last layer of the decoder but also at intermediate
layers. These predictions range from low to high-resolution, and are combined
into a single segmentation only at the end of the forward pass, in order to
compare it to the ground-truth. This multi-scale approach is shown to
effectively re-use information from all layers of the decoder, and results in
increased performance.

###  2.4 Training Details

All the models trained in this work follow the same protocol. We optimize
network weights to minimize the cross-entropy loss using standard Stochastic
Gradient Descent with a learning rate of $l=0.01$ and a batch-size of $4$. The
learning rate is decayed following a cosine law from its initial value to
$l=1e-8$ during $25$ epochs, which defines a training cycle. We then repeat
this process for $20$ cycles, restarting the learning back at the beginning of
each cycle. Images are re-scaled to $640\times 512$, which respects the
predominant rectangular aspect ratio in most polyp segmentation datasets, and
during training they are augmented with standard techniques (random rotations,
vertical/horizontal flipping, contrast/saturation/brightness changes). The
mean Dice score is monitored on a separate validation set and the best
performing model is kept for testing purposes. For testing, we generate four
different versions of each image by horizontal/vertical flipping, predict on
each of them, and average the results.

Dataset | Characteristics & Challenges
---|---
KVasir-Seg [19] |  Large scale (n=1000), diverse, includes multi-polyp cases Varying resolution from $487\times 332$ to $1920\times 1072$ pixels
CVC-ClinicDB [5] |  n=612 images of resolution $388\times 284$ from 31 sequences Different polyp categories and sizes, substantial specularities
CVC-ColonDB [4] |  n=300 images from 15 short sequences, $574\times 500$ resolution Great variability in types of polyp and appearances
ETIS-LaribPolypDB [29] |  n=196 still images with a resolution of $1225\times 966$ Contains 44 different polyps from 34 sequences
EndoScene [34] |  n=912, combination of CVC-ColonDB + CVC-ClinicDB Images of varying resolutions, only part of the test set used
Table 1: Description of each of the datasets considered in this paper. The
first two are employed for training purposes. We use the five of them to
assess performance on images similar/different to the training set.

##  3 Experimental Results

This section describes our experimental analysis on the performance of the
proposed approach to polyp segmentation. We first describe the data and the
performance metrics used to validate our technique. We then develop an
analysis of the accuracy of different combinations of encoders and decoders,
comparing the performance on the Kvasir dataset [19] of a standard encoder-
decoder network with that of its double encoder-decoder counterpart. Next we
re-train the best model in a large-scale experiment and comprehensively
analyze its performance when compared to other recent polyp segmentation
methods on five different databases.

###  3.1 Data and Evaluation Metrics

It is certainly complicated to compare different approaches for polyp
segmentation, as no clear evaluation protocol has emerged from the literature.
This would involve clearly defining training and test subsets from the
different publicly available datasets, as well as defining proper evaluation
metrics.

In order to provide a fair comparison, we propose the following experimental
setup. For the first set of experiments, we consider the KVasir-seg database
[19], which we split into 90% training and 10% test images. In this dataset,
we carry out ablation experiments to find out which is the best encoder-
decoder combination, and to demonstrate that double encoder-decoders offer
superior performance. For the second part of this section, where we compare
our approach against the state-of-the-art in a large-scale experiment, we
follow closely the data source definitions provided in [11]. This involves up
to five different databases that are described in Table 1. The considered
training set is the union of two subsets of KVasir-seg [19] and CVC-Clinic-DB
[5], containing 90% of the data. From this training set, a validation set is
separated before training is carried out. To evaluate the performance of our
models, we use the remaining 10% of both datasets as test sets, which allow us
to assess performance when test data is similar to training data. In addition,
we use the three other datasets shown in Table 1 to assess performance on data
that comes from a different source: CVC-ColonDB [4], ETIS-LaribDB [29],
EndoScene [34]. Note that the training/test splits are exactly the same111All
data, training/test splits, and results of other compared methods were
directly downloaded from https://github.com/DengPingFan/PraNet as in [11]. In
particular, we only use the part of the EndoScene test set that does not
belong to CVC-ClinicDB, which is used for training, referred to as CVC-300
below. All other performance results are also extracted from [11], which
ensures a correct comparison of all approaches.

Regarding performance metrics, a consensus has not been reached yet in the
literature [32]. We therefore follow the recommendation in [19] and compute
Dice and Intersection over Union (IoU) scores on each binary prediction of a
given test set, and report their averages across all images. Note that this is
different from [11], where the probabilistic (grayscale) predictions are used
to derive an overall Dice and IoU score: if predictions are considered to be
discretized in 8-bits, for each threshold in a predefined range
$\\{0,1,...,255\\}$ a score is computed. Then the mean score is kept for each
image, and the average of these is reported as the final performance. This is
different from conventional performance metrics reported in other papers, and
we prefer to follow the simpler approach of employing binary predictions to
compute a single Dice and IoU score, as the binary segmentation is the most
relevant outcome of a polyp segmentation pipeline, and it does not appear to
be suitable to test a method on thresholds that are never going to be used in
clinical practice. Nevertheless, as in [11], in order to also assess the
quality of the probabilistic predictions, we report Mean Absolute Error
computed as the average of the absolute value of pixelwise differences between
the binary ground-truth and the grayscale predicted segmentation.

In order to generate binary segmentations from probabilistic predictions and
calculate Dice/IOU scores, we use a simple adaptive thresholding algorithm
[26], which is a popular approach in polyp segmentation [2], followed by
morphological hole filling. The same binarization scheme is applied to the
grayscale predictions of the compare methods, which were provided by the
authors of [11].

###  3.2 Autoencoders vs Double Autoencoders

Decoder $\boldsymbol{\rightarrow}$ |  U-Net [27] |  DeepLab [9] |  FP-Net [24]
---|---|---|---
Encoder $\boldsymbol{\downarrow}$ | DICE | IOU | MAE | DICE | IOU | MAE | DICE | IOU | MAE
MobileNet [28] | 88.59 | 82.87 | 3.65 | 88.87 | 82.88 | 3.79 | 89.15 | 83.14 | 3.61
MobileNet$\times 2$ | 88.80 | 83.24 | 3.68 | 90.23 | 84.80 | 3.59 | 90.73 | 85.53 | 3.17
[2pt/5pt] Perf. Diff. | +0.21 | +0.37 |  +0.03 | +1.36 | +1.96 | -0.20 | +1.58 | +2.39 | -0.44
Resnet34 [16] | 89.72 | 84.28 | 3.24 | 89.26 | 83.66 | 3.31 | 89.90 | 84.76 | 3.00
Resnet34$\times 2$ | 90.13 | 84.77 | 3.09 | 90.39 | 85.30 | 3.12 | 90.70 | 85.55 | 3.01
[2pt/5pt] Perf. Diff. | +0.41 | +0.49 | -0.15 | +1.13 | +1.64 | -0.19 | +0.80 | +0.79 |  +0.01
DPN [10] | 89.72 | 83.91 | 3.19 | 90.23 | 84.83 | 3.25 | 89.96 | 84.61 | 3.04
DPN$\times 2$ | 90.21 | 85.10 | 3.25 | 91.21 | 86.16 | 2.84 | 91.97 | 87.05 | 2.65
[2pt/5pt] Perf. Diff. | +0.49 | +1.19 |  +0.06 | +0.98 | +1.33 | -0.39 | +2.01 | +2.44 | -0.39
Table 2: Performance analysis of different combinations of pretrained encoders
and decoders on the Kvasir databaset. Bold numbers indicate performance
improvements of double encoder-decoders upon their single encoder-decoder
counterparts.

We first train each of the combinations of pretrained encoders and decoders
that we described in Sections 2.2 and 2.3 on the K-Vasir dataset. Our aim with
this set of experiments is to investigate if using double encoder-decoders
brings performance improvements.

Table 2 reports the results of our experiments with different encoders and
decoders. The main pattern arising is that double encoder-decoders do provide
a performance boost in every considered case. Although the improvement is
varying, there is no single case in which single encoder-decoder networks
outperform their doubled counterparts. In addition, we can see that Feature-
Pyramid networks (FP-Net) are the best decoder for this particular problem,
surpassing the performance of U-Net and DeepLab for whichever considered
encoder network. For the particular case of FP-Net encoders, it is also
interesting to note that the lightweight MobileNet encoder achieves a better
performance when compared to the heavier ResNet34 encoder. It is also worth
noting that Dual Path Networks (DPN) stand out as the best pretrained encoder
for all the considered decoders. In the remaining of this section, we select
the DPN+FP-Net combination, and evaluate the performance of a double encoder-
decoder network against other recent polyp segmentation techniques.

###  3.3 Comparison with Recent Techniques

In this section we re-train the best double encoder-decoder network (referred
to as DPN68$\times 2$) on the dataset employed in [11] following the same
train/test split. This training set derived from the Kvasir and CVC-Clinic
databases is used to optimize the double DPN68$\times 2$ architecture, and we
conduct two types of experiments. First, we test the resulting model on data
similar to the training distribution by computing performance (separately) on
the KVasir and the CVC-Clinic test sets. Second, we use the ColonDB and ETIS
databases, which contain relatively dissimilar colonoscopic frames, to assess
the generalization ability of our approach. Following [11], we also include in
this experiment the CVC-300 database, although this dataset contains part of
the test set of CVC-Clinic, and it represents therefore an easier problem.

In both cases, we compare the performance of our technique with a) two popular
medical image segmentation architectures, namely U-Net [27] and its extensions
U-Net++ [39], and b) three architectures specifically developed for polyp
segmentation techniques, ResUnet++ [20], SFA [12], and PraNet [11].

| KVasir | CVC-Clinic
---|---|---
| DICE | IOU | MAE | DICE | IOU | MAE
U-Net / MICCAI’15 [27] | 83.00 | 75.91 | 5.47 | 84.39 | 77.70 | 1.92
ResUNet-mod[20] | 79.09 | 42.87 | —– | 77.88 | 45.45 | —–
ResUNet++ [20] | 81.33 | 79.27 | —– | 79.55 | 79.62 | —–
SFA / MICCAI’19 [12] | 73.10 | 61.87 | 7.54 | 70.61 | 61.26 | 4.16
U-Net++ / TMI’20 [39] | 83.21 | 75.60 | 14.95 | 81.20 | 74.67 | 7.97
PraNet / MICCAI’20 [11] | 90.37 | 84.79 | 2.96 | 90.72 | 85.89 | 0.93
DPN68$\times 2$ | 91.70 | 86.74 | 2.68 | 91.61 | 86.65 | 1.42
Table 3: Performance of the best double encoder-decoder model (DPN68$\times
2$) compared to other approaches on the test set of the KVasir-Seg and CVC-
Clinic databases. Models were learned on the combination of the training set
of both datasets.

Results of the first of our experiments are reported in Table 3. It can be
seen that the DPN68$\times 2$ architecture improves the performance of all
other approaches, achieving top performance by a wide margin with respect to
most considered methods. The second best approach is the recently introduced
PraNet CNN, which attains a performance relatively close to the one of
DPN68$\times 2$, although still slightly inferior.

It is however on our second experiment that reveals most remarkable
performance differences between our approach and PraNet and the other
considered techniques. The results displayed in Table 4 show that the
generalization capability of our DPN68$\times 2$ model is much larger than
previous approaches: when compared to the second best model (again PraNet), it
attains an increase 5.29 in DICE score in the CVC-ColonDB and 12.07 Dice score
in the ETIS databases. This margin is even larger for other methods, which
clearly signals to DPN68$\times 2$ being substantially more competitive when
using unseen data that is far away from the training distribution. On the
CVC-300 database, which is more similar to the training data, DPN68$\times 2$
still surpasses all other methods, although with a smaller difference.

| ColonDB | ETIS | CVC-300
---|---|---|---
| DICE | IOU | MAE | DICE | IOU | MAE | DICE | IOU | MAE
U-Net / MICCAI’15 | 53.91 | 46.28 | 5.86 | 44.02 | 36.72 | 3.62 | 73.99 | 65.30 | 2.21
SFA / MICCAI’19 | 45.90 | 33.83 | 7.46 | 29.88 | 21.82 | 4.55 | 46.90 | 32.95 | 6.51
U-Net++ / TMI’20 | 51.73 | 43.37 | 7.42 | 48.05 | 40.15 | 4.52 | 74.79 | 65.89 | 3.36
PraNet / MICCAI’20 | 72.42 | 65.11 | 4.31 | 64.12 | 57.98 | 3.11 | 87.63 | 80.35 | 0.99
DPN68$\times 2$ | 77.71 | 69.81 | 3.97 | 76.19 | 68.43 | 1.68 | 89.15 | 82.56 | 0.90
Table 4: Performance of the best double encoder-decoder model (DPN68$\times
2$) compared to other approaches on the ColonDB, ETIS, and CVC-300 databases.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 3: (a), (e): Polyps extracted from the Kvasir-Seg and CVC-Clinic
databases. Segmentations produced by (b), (f): PraNet, (c), (g): DPN68$\times
2$. (d), (h): Ground-truth

###  3.4 Qualitative Analysis

In this section, we visually analyze some of the segmentation results obtained
on the different test data when compared with other approaches.

Fig. 3 shows two challenging cases of polyps extracted from the Kvasir and
CVC-Clinic databases (test set associated to the training set), while Fig. 4
displays two polyps extracted from the test set of ColonDB and ETIS
(dissimilar data). We also show the results produced by the second best
technique, PraNet.

In the Kvasir example, it can be appreciated that DPN68$\times 2$ succeeds to
properly segment the polyp appearing in the center of the frame in Fig. (3a),
whereas PraNet is probably confused by the specularities present in the image
and oversegments this example, as shown in Figs. (3b) and (3c). As for the
CVC-Clinic example in Fig. (3e), the polyp appearing to the left of the frame
is reasonably segmented by both methods. However, the confounding artifact
above the polyp due to optical blurring is wrongly segmented by PraNet as part
of the anomaly, as seen in (Fig. 3f); DPN68$\times 2$ is also slightly
confused, but it assigns much less probability to the pixels pixels related to
the blurring artifact in Fig. (3g).

A challenging example extracted from the CVC-ColonDB is shown in the top row
of Fig. 4. This low-contrast frame leads to a wrong prediction in the PraNet
case shown in Fig. (4b), whereas the DPN68$\times 2$ model has no trouble in
accurately delineating the borders of the polyp in Fig. (4c). Finally, an
example from the ETIS database is shown in the bottom row of Fig. 4. This is a
hardly visible polyp, and PraNet erroneously locates it to the right of the
image, while DPN68$\times 2$, even if producing some oversegmenting, avoids
this confusion.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 4: (a), (e): Polyps extracted from the CVC-ColonDB and ETIS databases.
Segmentations produced by (b), (f): PraNet, (c), (g): DPN68$\times 2$. (d),
(h): Ground-truth

##  4 Discussion and Conclusion

In this work we have analyzed double decoder-encoder networks for the task of
polyp segmentation on endoscopic images. The encoder of the first network
generates an attention map that indicates the second network the most
interesting areas of the image. Double encoder-decoders leverage state-of-the-
art encoder pretrained large natural image databases. We have shown that they
provide a clear advantage over their single encoder-decoder counterparts.
Comprehensive experimental results also show that a double encoder-decoder
with a DPN encoder and a FP-Net decoder attains state-of-the-art on common
benchmarks, with particularly good results on data that is visually dissimilar
to the training images. Future steps involve the investigation of more
efficient combination of the two building blocks of double encoder-decoders.
It is reasonable to expect that smaller networks can already generate a
suitable attention map that can be employed by the second network to achieve
similar results at a reduced computational load.

## References

  * [1] Ahn, S.B., Han, D.S., Bae, J.H., Byun, T.J., Kim, J.P., Eun, C.S.: The Miss Rate for Colorectal Adenoma Determined by Quality-Adjusted, Back-to-Back Colonoscopies. Gut and Liver 6(1), 64–70 (Jan 2012)
  * [2] Akbari, M., Mohrekesh, M., Nasr-Esfahani, E., Soroushmehr, S.R., Karimi, N., Samavi, S., Najarian, K.: Polyp Segmentation in Colonoscopy Images Using Fully Convolutional Network. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp. 69–72 (Jul 2018), iSSN: 1558-4615
  * [3] Badrinarayanan, V., Kendall, A., Cipolla, R.: SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(12), 2481–2495 (Dec 2017)
  * [4] Bernal, J., Sánchez, J., Vilariño, F.: Towards automatic polyp detection with a polyp appearance model. Pattern Recognition 45(9), 3166–3182 (Sep 2012)
  * [5] Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., Vilariño, F.: WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics 43, 99–111 (Jul 2015)
  * [6] Bernal, J., Tajkbaksh, N., Sánchez, F.J., Matuszewski, B.J., Chen, H., Yu, L., Angermann, Q., Romain, O., Rustad, B., Balasingham, I., Pogorelov, K., Choi, S., Debard, Q., Maier-Hein, L., Speidel, S., Stoyanov, D., Brandao, P., Córdova, H., Sánchez-Montes, C., Gurudu, S.R., Fernández-Esparrach, G., Dray, X., Liang, J., Histace, A.: Comparative Validation of Polyp Detection Methods in Video Colonoscopy: Results From the MICCAI 2015 Endoscopic Vision Challenge. IEEE Transactions on Medical Imaging 36(6), 1231–1249 (Jun 2017)
  * [7] Carneiro, G., Zorron Cheng Tao Pu, L., Singh, R., Burt, A.: Deep learning uncertainty and confidence calibration for the five-class polyp classification from colonoscopy. Medical Image Analysis 62, 101653 (May 2020)
  * [8] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(4), 834–848 (Apr 2018)
  * [9] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv:1706.05587 [cs] (Dec 2017), arXiv: 1706.05587
  * [10] Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., Feng, J.: Dual path networks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. pp. 4470–4478. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (Dec 2017)
  * [11] Fan, D.P., Ji, G.P., Zhou, T., Chen, G., Fu, H., Shen, J., Shao, L.: PraNet: Parallel Reverse Attention Network for Polyp Segmentation. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 (2020)
  * [12] Fang, Y., Chen, C., Yuan, Y., Tong, K.y.: Selective Feature Aggregation Network with Area-Boundary Constraints for Polyp Segmentation. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 2019. pp. 302–310. Lecture Notes in Computer Science, Springer International Publishing, Cham (2019)
  * [13] Galdran, A., Anjos, A., Dolz, J., Chakor, H., Lombaert, H., Ayed, I.B.: The Little W-Net That Could: State-of-the-Art Retinal Vessel Segmentation with Minimalistic Models. arXiv:2009.01907 (Sep 2020)
  * [14] Guo, Y., Bernal, J., J. Matuszewski, B.: Polyp Segmentation with Fully Convolutional Deep Neural Networks—Extended Evaluation Study. Journal of Imaging 6(7), 69 (Jul 2020)
  * [15] Haggar, F.A., Boushey, R.P.: Colorectal Cancer Epidemiology: Incidence, Mortality, Survival, and Risk Factors. Clinics in Colon and Rectal Surgery 22(4), 191–197 (Nov 2009)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (Jun 2016), iSSN: 1063-6919
  * [17] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely Connected Convolutional Networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2261–2269 (Jul 2017), iSSN: 1063-6919
  * [18] Jha, D., Riegler, M., Johansen, D., Halvorsen, P., Johansen, H.D.: DoubleU-Net: A Deep Convolutional Neural Network for Medical Image Segmentation. 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS) (2020)
  * [19] Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., de Lange, T., Johansen, D., Johansen, H.D.: Kvasir-seg: A segmented polyp dataset. In: International Conference on Multimedia Modeling. pp. 451–462. Springer (2020)
  * [20] Jha, D., Smedsrud, P.H., Riegler, M.A., Johansen, D., Lange, T.D., Halvorsen, P., D. Johansen, H.: ResUNet++: An Advanced Architecture for Medical Image Segmentation. In: 2019 IEEE International Symposium on Multimedia (ISM). pp. 225–2255 (Dec 2019)
  * [21] Jia, X., Mai, X., Cui, Y., Yuan, Y., Xing, X., Seo, H., Xing, L., Meng, M.Q.H.: Automatic Polyp Recognition in Colonoscopy Images Using Deep Learning and Two-Stage Pyramidal Feature Prediction. IEEE Transactions on Automation Science and Engineering 17(3), 1570–1584 (Jul 2020)
  * [22] Kang, J., Gwak, J.: Ensemble of Instance Segmentation Models for Polyp Segmentation in Colonoscopy Images. IEEE Access 7, 26440–26447 (2019)
  * [23] Li, L., Verma, M., Nakashima, Y., Nagahara, H., Kawasaki, R.: IterNet: Retinal Image Segmentation Utilizing Structural Redundancy in Vessel Networks. In: The IEEE Winter Conference on Applications of Computer Vision (WACV) (Mar 2020)
  * [24] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature Pyramid Networks for Object Detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 936–944 (Jul 2017), iSSN: 1063-6919
  * [25] Lui, T.K., Hui, C.K., Tsui, V.W., Cheung, K.S., Ko, M.K., aCC Foo, D., Mak, L.Y., Yeung, C.K., Lui, T.H., Wong, S.Y., Leung, W.K.: New insights on missed colonic lesions during colonoscopy through artificial intelligence–assisted real-time detection (with video). Gastrointestinal Endoscopy (May 2020)
  * [26] Otsu, N.: A Threshold Selection Method from Gray-Level Histograms. IEEE Transactions on Systems, Man, and Cybernetics 9(1), 62–66 (Jan 1979)
  * [27] Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. pp. 234–241. Lecture Notes in Computer Science, Springer International Publishing, Cham (2015)
  * [28] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: Inverted Residuals and Linear Bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4510–4520 (Jun 2018)
  * [29] Silva, J., Histace, A., Romain, O., Dray, X., Granado, B.: Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer. International Journal of Computer Assisted Radiology and Surgery 9(2), 283–293 (Mar 2014)
  * [30] Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. In: International Conference on Learning Representations (2015)
  * [31] Sánchez-Peralta, L.F., Bote-Curiel, L., Picón, A., Sánchez-Margallo, F.M., Pagador, J.B.: Deep learning to find colorectal polyps in colonoscopy: A systematic literature review. Artificial Intelligence in Medicine 108, 101923 (Aug 2020)
  * [32] Sánchez-Peralta, L.F., Picón, A., Antequera-Barroso, J.A., Ortega-Morán, J.F., Sánchez-Margallo, F.M., Pagador, J.B.: Eigenloss: Combined PCA-Based Loss Function for Polyp Segmentation. Mathematics 8(8), 1316 (Aug 2020)
  * [33] Tian, Z., He, T., Shen, C., Yan, Y.: Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation. pp. 3126–3135 (2019)
  * [34] Vázquez, D., Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., López, A.M., Romero, A., Drozdzal, M., Courville, A.: A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images. Journal of Healthcare Engineering 2017, 4037190 (Jul 2017)
  * [35] Wickstrøm, K., Kampffmeyer, M., Jenssen, R.: Uncertainty and interpretability in convolutional neural networks for semantic segmentation of colorectal polyps. Medical Image Analysis 60, 101619 (Feb 2020)
  * [36] Wojna, Z., Ferrari, V., Guadarrama, S., Silberman, N., Chen, L.C., Fathi, A., Uijlings, J.: The Devil is in the Decoder: Classification, Regression and GANs. International Journal of Computer Vision 127(11), 1694–1706 (Dec 2019)
  * [37] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated Residual Transformations for Deep Neural Networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5987–5995 (Jul 2017), iSSN: 1063-6919
  * [38] Zhang, R., Zheng, Y., Poon, C.C.Y., Shen, D., Lau, J.Y.W.: Polyp detection during colonoscopy using a regression-based convolutional neural network with a tracker. Pattern Recognition 83, 209–219 (Nov 2018)
  * [39] Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N., Liang, J.: UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation. IEEE Transactions on Medical Imaging 39(6), 1856–1867 (Jun 2020)