2022

1]Research School of Engineering, Australian National University, Canberra,
Australia

2]College of Computer and Control Engineering, Minjiang University, Fuzhou,
China

3]Department of Computer Science, Johns Hopkins University, Baltimore, USA

4]Computer Vision Lab, ETH Zürich, Zürich, Switzerland

5]Department of Radiological Sciences, UCLA, Los Angeles, USA

6]School of Computer Science and Engineering, NPU, Xi’an, China

]Project Page: https://github.com/GewelsJI/VPS

# Video Polyp Segmentation: A Deep Learning Perspective

Ge-Peng Ji$\dagger$ Guobao Xiao$\dagger$ Yu-Cheng Chou$\dagger$ Deng-Ping Fan🖂
Kai Zhao  Geng Chen  Luc Van Gool  [  [  [  [  [  [  [

###### Abstract

We present the first comprehensive video polyp segmentation (VPS) study in the
deep learning era. Over the years, developments in VPS are not moving forward
with ease due to the lack of a large-scale dataset with fine-grained
segmentation annotations. To address this issue, we first introduce a high-
quality frame-by-frame annotated VPS dataset, named SUN-SEG, which contains
158,690 colonoscopy video frames from the well-known SUN-database. We provide
additional annotation covering diverse types, _i.e._ , attribute, object mask,
boundary, scribble, and polygon. Second, we design a simple but efficient
baseline, named PNS+, which consists of a global encoder, a local encoder, and
normalized self-attention (NS) blocks. The global and local encoders receive
an anchor frame and multiple successive frames to extract long-term and short-
term spatial-temporal representations, which are then progressively refined by
two NS blocks. Extensive experiments show that PNS+ achieves the best
performance and real-time inference speed (170fps), making it a promising
solution for the VPS task. Third, we extensively evaluate 13 representative
polyp/object segmentation models on our SUN-SEG dataset and provide attribute-
based comparisons. Finally, we discuss several open issues and suggest
possible research directions for the VPS community.

###### keywords:

Video polyp segmentation, dataset, self-attention, colonoscopy, abdomen.

##  1 Introduction

††† Equal contribution. 🖂 Corresponding author.

As the second most deadly cancer and the third most common malignancy,
colorectal cancer (CRC) is estimated to cause millions of incidence cases and
deaths yearly. The survival rate of CRC patients is over $95\%$ at the first
stage of the disease but dramatically decreases to lower than $35\%$ at the
fourth and fifth stages bernal2012towards . Therefore, early diagnosis of
positive CRC cases through screening techniques, such as colonoscopy and
sigmoidoscopy, is vital in increasing the survival rate. For prevention
purposes, physicians can remove the colon polyps that risk turning into
cancer. However, this process highly depends on the physicians’ experience and
suffers from a high polyp missing rate, _i.e._ , $22\%\sim 28\%$
puyal2020endoscopic .

Recently, artificial intelligence (AI) techniques are applied to the automatic
detection of candidate lesion polyps during colonoscopy for physicians.
However, developing AI models with a satisfactory detection rate is still
challenging due to two problems: (a) Limited Annotated Data. Deep learning
models are often hungry for a large-scale video dataset with densely-annotated
labels. Moreover, a community-agreed benchmark is missing for evaluating the
approaches’ actual performance. (b) Dynamic Complexity. The colonoscopy
usually involves less ideal conditions of camera-moving acquisition, such as
the diversity of colon polyps (_e.g._ , boundary contrast, shape, orientation,
shooting angle), internal artifacts (_e.g._ , water flow, residue), and
imaging degradation (_e.g._ , color distortion, specular reflection). To this
end, we present a systematic study to facilitate the development of deep
learning models for video polyp segmentation (VPS). The main contributions of
this work are summarized as follows:

  * •

VPS Dataset. We elaborately introduce a large-scale VPS dataset termed SUN-
SEG, containing $158,690$ frames selected from the SUN-database
misawa2020development . We provide a variety of labels, including attribute,
object mask, boundary, scribble, and polygon. These labels can further support
the development of colonoscopy diagnosis, localization, and derivative tasks.

  * •

VPS Baseline. We design a simple but efficient VPS baseline, named PNS+, which
consists of a global encoder, a local encoder, and two normalized self-
attention (NS) blocks. The global and local encoders extract long-and short-
term spatial-temporal representations from the first anchor frame and multiple
successive frames, respectively. The NS block dynamically updates the
receptive field when coupling attentive cues among extracted features.
Experiments show that PNS+ achieves the best performance on challenging SUN-
SEG dataset.

  * •

VPS Benchmark. To comprehensively understand VPS development, we conduct the
first large-scale benchmark by evaluating 13 cutting-edge polyp/object
segmentation approaches. Based on the benchmarking results (_i.e._ , five
image-based and eight video-based), we argue that the VPS task is not well
undertaken and leaves plenty of room for further exploration.

A preliminary version of this work was presented in ji2021pnsnet . In this
extended work, we introduce three different contributions.

  * •

In Sec. 3, we introduce a high-quality densely-annotated VPS dataset, SUN-SEG,
with five extended labels, _i.e._ , attribute, object mask, boundary,
scribble, and polygon.

  * •

Based on the normalized self-attention block as in ji2021pnsnet , we propose a
global-to-local learning paradigm to realize the modeling of both long-term
and short-term dependencies. This part is detailed in Sec. 4.3.

  * •

As shown in Sec. 5, we construct the first large-scale benchmark on the VPS
task, which contains $13$ the latest polyp/object segmentation competitors. We
highlight several potential research directions based on the above benchmark
results and progress in the VPS field.

##  2 Related Works

This section reviews the recent efforts in computer-aided polyp diagnosis from
the following two aspects: colonoscopy-related datasets (Sec. 2.1) and
approaches (Sec. 2.2).

###  2.1 Colonoscopy-Related Datasets

Recently, several datasets have been collected for the examination of human
colonoscopy. As shown in Table 1, we summarize some key statistics of $20$
popular datasets and our SUN-SEG dataset. In light of the task definition, we
categorize them into three main-stream partitions.

Table 1: Statistics of existing 20 datasets for human colonoscopy. #IMG = number of images. #VID = number of video sequences. DL = densely labeling. CLS = classification label. BBX = bounding box. PM = pixel-level mask. DATASET | YEAR | #IMG | #VID | DL | CLS | BBX | PM | Website
---|---|---|---|---|---|---|---|---
CVC-ColonDB bernal2012towards  | 2012 | 300 | 13 |  |  |  | ✓ | http://mv.cvc.uab.es/projects/colon-qa/cvccolondb
ETIS-Larib silva2014toward  | 2014 | 196 | 34 |  |  |  | ✓ | https://polyp.grand-challenge.org/EtisLarib/
CVC-ClinicDB bernal2015wm  | 2015 | 612 | 31 |  |  |  | ✓ | https://polyp.grand-challenge.org/CVCClinicDB/
ColonoscopicDS mesejo2016computer  | 2016 | - | 76 |  | ✓ |  |  | http://www.depeca.uah.es/colonoscopy_dataset/
ASU-Mayo tajbakhsh2015automated  | 2016 | 36,458 | 38 | ✓ |  |  | ✓ | https://polyp.grand-challenge.org/AsuMayo/
CVC-ClinicVideoDB giana2017  | 2017 | 11,954 | 18 | ✓ |  | ✓ |  | https://endovissub2017-giana.grand-challenge.org/Home/
CVC-EndoSceneStill vazquez2017benchmark  | 2017 | 912 | 44 |  |  |  | ✓ | http://www.cvc.uab.es/CVC-Colon/index.php/databases/cvc-endoscenestill/
KID2 koulaouzidis2017kid ; iakovidis2018detecting  | 2017 | 2,371 | 47 |  | ✓ |  | ✓ | https://mdss.uth.gr/datasets/endoscopy/kid/
Kvasir pogorelov2017kvasir  | 2017 | 8,000 | - |  | ✓ |  |  | https://datasets.simula.no/kvasir/
EDD2020 ali2020endoscopy  | 2020 | 386 | - |  | ✓ | ✓ | ✓ | https://edd2020.grand-challenge.org/
SUN-database misawa2020development  | 2020 | 158,690 | 113 | ✓ | ✓ | ✓ |  | http://amed8k.sundatabase.org/
Hyper-Kvasir borgli2020hyperkvasir  | 2020 | 110,079 | 374 |  | ✓ | ✓ | ✓ | https://datasets.simula.no/hyper-kvasir/
Kvasir-SEG jha2020kvasir  | 2020 | 1,000 | - |  |  |  | ✓ | https://datasets.simula.no/kvasir-seg/
PICCOLO sanchez2020piccolo  | 2020 | 3,433 | 40 |  | ✓ |  | ✓ | https://www.biobancovasco.org/en/Sample-and-data-catalog/Databases/PD178-PICCOLO-EN.html
Kvasir-Capsule smedsrud2021kvasir  | 2021 | 4,741,504 | 117 | ✓ | ✓ | ✓ |  | https://github.com/simula/kvasir-capsule
CP-CHILD-A wang2020improved  | 2021 | 8,000 | - |  | ✓ |  |  | https://figshare.com/articles/dataset/CP-CHILD_zip/12554042
CP-CHILD-B wang2020improved  | 2021 | 1,500 | - |  | ✓ |  |  | https://figshare.com/articles/dataset/CP-CHILD_zip/12554042
LDPolypVideo ma2021ldpolypvideo  | 2021 | 40,266 | 160 | ✓ | ✓ | ✓ |  | https://github.com/dashishi/LDPolypVideo-Benchmark
KUMC li2021colonoscopy  | 2021 | 37,899 | 155 | ✓ | ✓ | ✓ |  | https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/FCBUOR
PolypGen ali2021polypgen  | 2021 | 6,282 | 26 |  | ✓ | ✓ | ✓ | https://github.com/sharibox/PolypGen-Benchmark
SUN-SEG (OUR) | 2022 | 158,690 | 1,013 | ✓ | ✓ | ✓ | ✓ | https://github.com/GewelsJI/VPS

####  2.1.1 Classification

There are four popular datasets for the initial purpose of identifying
gastrointestinal lesions. ColonoscopicDS mesejo2016computer  collects $76$
regular colonoscopy videos with three types of gastrointestinal lesions,
including hyperplasic, serrated, and adenoma lesions. Kvasir
pogorelov2017kvasir  contains $8$ types of anatomical landmarks (_i.e._ ,
polyps, esophagitis, ulcerative colitis, z-line, pylorus, cecum, dyed polyp,
and dyed resection margins), and each type has $1,000$ images. Hyper-Kvasir
borgli2020hyperkvasir  further collects $110,079$ samples from $374$
colonoscopy videos, containing three types of annotations: $10,662$ class
labels with $23$ different lesion findings and $1,000$ images with segmented
masks and bounding box labels. Notably, all the segmented masks in Hyper-
Kvasir are selected from Kvasir-SEG borgli2020hyperkvasir . Recently, CP-
CHILD-A & -B wang2020improved  record the colonoscopy data from children,
including two classes (_i.e._ , colon polyp, normal or other pathological
images) for the classification task.

####  2.1.2 Detection

There are five widely-accepted video datasets mainly used for the detection
task. CVC-ClinicVideoDB giana2017 , as the early video dataset, comprises $18$
videos with a total number of $11,954$ frames in which $10,025$ frames contain
at least a polyp. As for the largest densely-annotated video polyp detection
dataset, SUN-database misawa2020development  consists of $49,136$ positive
samples with their bounding boxes acquired from $99$ patients. More recently,
two video datasets (_i.e._ , Kvasir-Capsule smedsrud2021kvasir  and KUMC
li2021colonoscopy ) are applied for both detection and classification tasks.
Especially, the former provides $47,238$ bounding box labels from $14$ lesion
classes, and the latter has $37,899$ frames with bounding box labels. Unlike
the above datasets, LDPolypVideo ma2021ldpolypvideo  includes $40,266$ frames
with circular annotations from $160$ colonoscopy videos.

####  2.1.3 Segmentation

As for the video datasets, the early benchmark CVC-EndoSceneStill
vazquez2017benchmark  opts for the combination of CVC-ColonDB
bernal2012towards  and CVC-ClinicDB bernal2015wm . ETIS-Larib silva2014toward
provides $196$ labeled samples from $32$ colonoscopy videos, containing about
five frames for each sequence. EDD2020 ali2020endoscopy  contains $386$
endoscopy images from five different institutions and multiple
gastrointestinal organs. They provide annotations for disease detection,
localization, and segmentation. PICCOLO sanchez2020piccolo  also samples
$3,433$ frames from $40$ videos with sparse annotations. As such, the above
five video datasets adopt the sampling annotation strategy, which still lacks
per-frame masks on each video sequence due to the labor-intensive annotation
process. Being the pioneering video dataset with densely-annotated masks, ASU-
Mayo tajbakhsh2015automated  contains $36,458$ continuous frames from $38$
videos, while it only provides $3,856$ labels for $10$ positive videos.
Recently, PolypGen ali2021polypgen  has collected a multi-centre dataset
incorporating more than $300$ patients, including single and continuous frames
with $3,788$ annotated segmentation masks and bounding box labels. Unlike
existing works, we introduce SUN-SEG, the first high-quality densely-annotated
dataset for the VPS task, which contains rich annotated labels, such as object
mask, boundary, scribble, polygon, and attribute. We hope that this work could
fuel the development of colonoscopy diagnosis, localization, and derivative
tasks.

###  2.2 Colonoscopy-Related Methods

Early solutions bernal2012towards ; dhandra2006analysis ; mamonov2014automated
; maghsoudi2017superpixel  have been dedicated to identifying colon polyps via
mining hand-crafted patterns, such as color, shape, texture, and super-pixel.
However, they usually suffer from low accuracy due to the limited capability
of representing heterogeneous polyps, as well as the close resemblance between
polyps and hard mimics yu2016integrating . In contrast, data-driven AI
techniques can handle these challenging conditions with better learning
ability. This section mainly focuses on tracking the latest image/video polyp
segmentation techniques tavanapong2022artificial , while leaving the
systematic review of polyp classification gammulle2020two ; carneiro2020deep
and detection zhang2018polyp ; wu2021multi  in our future works.

####  2.2.1 Image Polyp Segmentation (IPS)

Several methods have been proposed to locate the pixel-level polyp regions
from the colonoscopy images. They can be grouped into two major categories.
(a) CNN-based Approaches. Brandao et al. brandao2017fully  adopted a fully
convolutional network (FCN) with a pre-trained model to segment polyps. Later,
Akbari et al. akbari2018polyp  introduced a modified FCN to improve the
segmentation accuracy. Inspired by the vast success of UNet ronneberger2015u
in biomedical image segmentation, UNet++ zhou2018unetplus  and ResUNet
jha2019resunetplus  were employed for polyp segmentation for improved
performance. Furthermore, PolypSeg zhong2020polypseg , ACS zhang2020adaptive ,
ColonSegNet jha2021real  and SCR-Net wu2021precise  explore the effectiveness
of UNet-enhanced architecture on adaptively learning semantic contexts. As the
newly-proposed methods, SANet wei2021shallow  and MSNet zhao2021automatic
design the shallow attention module and subtraction unit, respectively, to
achieve precise and efficient segmentation. Additionally, several works opt
for introducing additional constraints via three main-stream manners: exerting
explicit boundary supervision murugesan2019psi ; wang2022boundary ;
fang2019selective ; shen2021hrenet ; ji2022eernet , introducing implicit
boundary-aware representation fan2020pra ; nguyen2021ccbanet ;
cheng2021learnable , and exploring uncertainty for the ambiguous regions
kim2021uacanet . (b) Transformer-based Approaches. Recently, Transformers
shamshad2022transformers  have been gaining popularity thanks to their
powerful modeling ability. TransFuse zhang2021transfuse  combines the
Transformer and CNN, termed the parallel-in-branch scheme, for capturing
global dependencies and low-level spatial details. Besides, A BiFusion module
was designed to fuse multi-level features from both branches. Segtran
li2021medical  proposes a squeezed attention block that regularizes self-
attention, and the expansion block learns diversified representations. A
positional encoding scheme was proposed to impose an inductive continuity
bias. Based on PVT wang2021pvtv2 , Dong et al. dong2021polyp  introduced a
model with three tight components, _i.e._ , cascaded fusion, camouflage
identification, and similarity aggregation modules.

####  2.2.2 Video Polyp Segmentation (VPS)

Despite their progress, existing IPS methods suffer from an inherent
limitation of overlooking the valuable temporal cues in the colonoscopy
videos. Therefore, efforts have been dedicated to combining spatial-temporal
features among consecutive video frames. A hybrid 2/3D CNN framework
puyal2020endoscopic  was proposed to aggregate spatial-temporal correlations
and achieves better segmentation results. However, the kernel size restricts
the spatial correlation between frames, restricting the accurate segmentation
of fast movements of polyps. To alleviate the above problem, PNSNet
ji2021pnsnet  introduces a normalized self-attention (NS) block to learn
spatial-temporal representations with neighborhood correlations effectively.
In this paper, we delve deeper into a more effective global-to-local learning
strategy based on NS block, which can fully leverage both long-term and short-
term spatial-temporal dependencies.

##  3 VPS Dataset

We describe the introduced SUN-SEG dataset’s details in terms of data
collection/re-organization (Sec. 3.1), professional annotations (Sec. 3.2),
and dataset statistics (Sec. 3.3).

###  3.1 Data Organization

The colonoscopy videos in our SUN-SEG are from Showa University and Nagoya
University database (also named SUN-database misawa2020development ), the
largest video polyp dataset for the detection task. There are two advantages
of adopting the SUN-database as our data source. (a) Challenging Scenarios:
The videos are captured by the high-definition endoscope (CF-HQ290ZI &
CF-H290ECI, Olympus) and video recorder (IMH-10, Olympus), providing videos of
various polyp sizes at dynamic scenarios, such as imaging at different
focusing distances and speeds. (b) Reliable Pathological Localization: The
initial classification information and bounding box annotations are provided
by three research assistants and examined by two expert endoscopists with
professional domain knowledge.

(a) RedundantAnnotation(b) WaterOcclusion

Figure 1: High-criteria control for data annotation. For instance, we reject
case (a), where the boundary is not consistent with the polyp, and case (b),
where the water overlapping area is falsely annotated.

The origin SUN-database has $113$ colonoscopy videos, including $100$ positive
cases with $49,136$ polyp frames and $13$ negative cases with $109,554$ non-
polyp frames111These statistic data come from this
http://amed8k.sundatabase.org/, which is different from the data reported in
the original paper misawa2020development . Besides, the SUN-database is
available for only non-commercial use in research or educational purpose,
which could be freely accessed with permission from authors.. We manually trim
them into $378$ positive and $728$ negative clips while maintaining their
consecutive intrinsic relationship. Such data pre-processing ensures that each
clip has around 3$\sim$11s duration at a real-time frame rate (_i.e._ , $30$
fps), promoting the fault-tolerant margin for various algorithms and devices.
To this end, the re-organized SUN-SEG contains $1,106$ short video clips with
$158,690$ video frames totally, offering a solid foundation to build such a
representative benchmark.

###  3.2 Professional Annotations

Following fan2020camouflaged , we adopt a similar annotation pipeline.
According to the origin bounding box labels of the SUN-database
misawa2020development , ten experienced annotators are instructed to offer
various labels using Adobe Photoshop. Then, three colonoscopy-related
researchers re-verify the quality and correctness of these initial
annotations. Fig. 1 shows two typical samples under the restricted quality
controls (_i.e._ , rejected and passed). In addition to the original
pathological materials provided by SUN-database, such as pathological pattern
(_e.g._ , low-grade adenoma, hyperplastic polyp, _etc_), shape (_e.g._ ,
pedunculated, subpedunculated, _etc_), and location (_e.g._ , cecum, ascending
colon, _etc_), we further extend them with diversified annotations in our SUN-
SEG. The newly-extended annotations consist of the following five hierarchies:
visual attribute $\rightarrow$ object mask $\rightarrow$ boundary
$\rightarrow$ scribble $\rightarrow$ polygon. Selected samples and
corresponding annotations could be found in Fig. 2 and their
illustrations222The descriptions of complete annotations refer to
https://github.com/GewelsJI/VPS/blob/main/docs/DATA_DESCRIPTION.md. are as
follows.

Figure 2: Diversified annotations for each video frame in our SUN-SEG dataset including object mask (a), boundary (b), and two weak labels, _i.e._ , scribble (c) and polygon (d). More details refer to Sec. 3.2.  Table 2: List of ten types of visual attributes (ATTR.) and their descriptions. ATTR. | DESCRIPTION
---|---
SI |  _Surgical Instruments._ The endoscopic surgical procedures involve the positioning of instruments, such as snares,
| forceps, knives, and electrodes.
IB |  _Indefinable Boundaries._ The foreground and background areas around the object have similar color.
HO |  _Heterogeneous Object._ Object regions have distinct colors.
GH |  _Ghosting._ Object has anomaly RGB-colored boundary due to fast moving or insufficient refresh rate.
FM |  _Fast-motion._ The average per-frame object motion in a clip, computed as the Euclidean distance of polyp centroids
|  between consecutive frames, is larger than $20$ pixels.
SO |  _Small Object._ The average ratio between the object size and the image area in a clip is smaller than $0.05$.
LO |  _Large Object._ The average ratio between the object size and the image area in a clip is larger than $0.15$.
OC |  _Occlusion._ Polyp object becomes partially or fully occluded.
OV |  _Out-of-view._ Polyp object is partially clipped by the image boundaries.
SV |  _Scale-variation._ The average area ratio among any pair of bounding boxes enclosing the target object in a clip is
|  smaller than $0.5$.

  * •

Visual Attribute. According to the visual characteristics of videos, we
provide ten visual attributes at the video level, whose classification
criteria are detailed in Table 2.

  * •

Object Mask. Correctly parsing lesion areas is helpful for a clinician.
Therefore, as shown in Fig. 2 (a), we provide pixel-wise object masks for each
frame. We further refine the coordinates of the original bounding box based on
the object mask to tighten the target, offering more reliable localization
labels.

  * •

Boundary. Fig. 2 (b) shows the polyp boundary generated by calculating the
gradient of the object mask.

  * •

Scribble. Besides, we offer two weak labels to facilitate the research under
data-insufficient conditions. As for the scribble labels in Fig. 2 (c), we use
two high-degree curves to indicate the foreground (purple curve) and
background (white curve), respectively. To ensure the objectivity of various
annotators, we adopt linear or quadratic functions to randomly create the
above curves in the positive/negative region.

  * •

Polygon. Similarly, in Fig. 2 (d), we randomly deploy the Douglas-Peucker
algorithm RAMER1972244  to find the circumscribed or inscribed polygons that
fit the object boundaries.

SUN-SEG-TrainCVC-300CVC-612SUN-SEG-HardSUN-SEG-Easy

Figure 3: The calculation of center bias fan2021salient  on CVC-300, CVC-612,
and our SUN-SEG-Train/-Easy/-Hard.

###  3.3 Dataset Statistics

This section discusses several vital statistics of our three SUN-SEG sub-
datasets for better illustration. More details about the data split of SUN-SEG
refer to Sec. 4.4.1.

  * •

Center Bias. Unlike general object detection, medical images usually share a
higher center bias since the targets are often not in the center of an image.
To depict the degree of center bias fan2021salient , we compute the average
distribution of each dataset’s overall ground truth map. Fig. 3 and Fig. 4
(top) show that the three sub-datasets of SUN-SEG have lower a center bias
than CVC-300 and CVC-612 datasets.

  * •

Polyp Size. Colonoscopy is an ego-motion situation instead of shooting moving
targets (_i.e._ , stuff and things) through fixed cameras in the general
domain. As a result, the scale variation of polyps and the irregular movement
of the camera causes the different sizes of polyps. The polyps partly or even
fully disappear in the view. Fig. 4 (bottom-left) shows the comparison of
polyp scales at five different VPS datasets.

  * •

Global/Local Contrast. To demonstrate how difficult a colon polyp is to
identify, in Fig. 4 (bottom-right), we describe it quantitatively using the
global and local contrast strategy fan2020rethinking .

SUN-SEG-TrainSUN-SEG-EasySUN-SEG-HardCVC-300CVC-612

Figure 4: Statistic curves among existing VPS datasets (CVC-300 & CVC-612) and
our SUN-SEG-Train/-Easy/-Hard. Note that the horizontal and vertical axes
denote the frequency and their statistic values, respectively. These curves
present the diversity of our dataset.

##  4 VPS Baseline

This section first clarifies the formulation of VPS task in Sec. 4.1. Then, we
describe the details of PNS+, including the normalized self-attention block
(Sec. 4.2), global-to-local learning strategy (Sec. 4.3), and implementation
details (Sec. 4.4).

\begin{overpic}[width=433.62pt]{./Imgs/GLNS_Framework_V5-min}
\put(18.5,4.5){$\scriptstyle\\{\mathcal{X}^{l}_{s}\\}_{s=t}^{t+\Delta}$}
\put(18.5,10.8){$\scriptstyle\\{\mathcal{X}^{h}_{s}\\}_{s=t}^{t+\Delta}$}
\put(31.0,8.3){$\scriptstyle\\{\mathcal{X}^{h}_{s}\\}_{s=t}^{t+\Delta}$}
\put(44.0,5.8){$\scriptstyle\\{\mathcal{X}^{h}_{s}\\}_{s=t}^{t+\Delta}$}
\put(43.25,26.0){\footnotesize Sec.~{}\ref{sec:glns_pipeline}} \end{overpic}
Figure 5: The pipeline of the proposed (a) PNS+ network, which is based on (b)
the normalized self-attention (NS) block.

###  4.1 Task Formulation

We mainly focus on the task of video polyp segmentation, which could be
defined as a binary-class video object segmentation task, _i.e._ , identifying
polyp and non-polyp areas. Specifically, our goal is to render a model to
assign a probability prediction (_i.e._ , a non-binary mask ranging from 0 to
1) for every pixel of each frame. Besides, we leave other types of tasks for
future exploration, such as video polyp detection.

###  4.2 Normalized Self-attention Block

Recently, the self-attention mechanism wang2018non  has been widely exploited
in many popular computer vision tasks. Our initial studies found that
introducing the original self-attention mechanism to the VPS task does not
achieve satisfactory results (high accuracy and speed) due to the multiscale
property of polyps that are captured at various shooting angles and speeds.
Directly utilizing the naive self-attention scheme, such as the non-local
network wang2018non , incurs a high computational cost, limiting the inference
speed. As shown in Fig. 5 (right), we propose a normalized self-attention (NS)
block, which is motivated by the fact that dynamically updating the receptive
field is important for self-attention-based networks. The NS block involves
five key steps, which are detailed as follows.

####  4.2.1 Enhanced Rules

Motivated by the recent video salient object detection model gu2020pyramid ,
we utilize three strategies, _i.e._ , channel split, query-dependent, and
normalization, to reduce the computational cost and improve the accuracy.

(a) Channel Split Rule. Specifically, given three candidate features (_i.e._ ,
query feature $Q$, key feature $K$, and value feature $V$) with the size of
$\mathbb{R}^{T\times H\times W\times C}$, we utilize three linear embedding
functions $\theta(\cdot)$, $\phi(\cdot)$, and $g(\cdot)$ to generate the
corresponding attention features. These functions can be implemented by a
convolutional layer with a kernel size of $1\times 1\times 1$ wang2018non .
Note that $T$, $H$, $W$ and $C$ denote the number of frames, height, width,
and channels of the given feature, respectively. This rule can be expressed
as:

$Q_{i}=\mathcal{F}^{G}\langle\theta(Q)\rangle,K_{i}=\mathcal{F}^{G}\langle\phi(K)\rangle,V_{i}=\mathcal{F}^{G}\langle
g(V)\rangle,$ (1)

where the function $\mathcal{F}^{G}$ denotes the operation that we split each
attention feature into $N$ groups along the channel dimension, resulting in
three disparate features: query $Q_{i}$, key $K_{i}$, and value $V_{i}$, where
$i=\\{1,2,\cdots,N\\}$. Thus, the shape of the above three split features is
$\mathbb{R}^{T\times H\times W\times\frac{C}{N}}$.

(b) Query-Dependent Rule. To model the spatial-temporal relationship among
consecutive frames, we need to measure the similarity between the split query
features $\\{Q_{i}\\}_{i=1}^{N}$ and split key features
$\\{K_{i}\\}_{i=1}^{N}$. Inspired by gu2020pyramid , we introduce $N$
relevance measuring (_i.e._ , query-dependent rule) blocks to compute the
spatial-temporal affinity matrix for the constrained neighborhood of the
target pixel. Rather than computing the response between a query position and
a key feature at all positions, as done in wang2018non , the relevance
measuring block can capture more relevance regarding the target object within
$T$ frames. More specifically, we get the corresponding constrained
neighborhood in $K_{i}$ for query pixel $\mathbf{X}^{q}$ of $Q_{i}$ in
position $(x,y,z)$, which can be obtained by a point sampling function
$\mathcal{F}^{S}$. This is formulated as:

$\mathcal{F}^{S}\langle\mathbf{X}^{q},K_{i}\rangle=\Sigma_{m=x-kd_{i}}^{x+kd_{i}}\Sigma_{n=y-kd_{i}}^{y+kd_{i}}\Sigma_{t=1}^{T}K_{i}(m,n,t),$
(2)

where $1\leq x\leq H$, $1\leq y\leq W$, and $1\leq z\leq T$ and
$\mathcal{F}^{S}\langle\mathbf{X}^{q},K_{i}\rangle\in\mathbb{R}^{T(2k+1)^{2}\times\frac{C}{N}}$.
Thus, the size of the constrained neighborhood will depend on the various
spatial-temporal receptive fields with different kernel sizes $k$, dilation
rate $d_{i}$ at the $i$-th group, and frame number $T$, respectively.

(c) Normalization Rule. However, the internal covariate shift problem
guo2020normalized  exists in the feed-forward of input $Q_{i}$, incurring that
the layer parameters cannot dynamically adapt to the next mini-batch. Thus, we
maintain a fixed distribution for $Q_{i}$ via:

$\hat{Q}_{i}=\texttt{Norm}(Q_{i}),$ (3)

where $\texttt{Norm}(\cdot)$ is implemented by layer normalization ba2016layer
operation along the temporal dimension.

####  4.2.2 Relevance Measuring

The affinity matrix $\mathcal{M}^{A}_{i}$ measures the similarity of target
pixels and their surrounding spatial-temporal contents in an adaptive point
sampling manner (refer to Equ.(2)). It is defined as:

$\mathcal{M}^{A}_{i}=\texttt{Softmax}(\frac{\hat{Q}_{i}\mathcal{F}^{S}\langle\hat{\mathbf{X}}^{q},K_{i}\rangle^{\top}}{\sqrt{C/N}}),~{}\text{when}~{}\hat{\mathbf{X}}^{q}\in\hat{Q}_{i},$
(4)

where $\mathcal{M}^{A}_{i}\in\mathbb{R}^{THW\times T(2k+1)^{2}}$. $\sqrt{C/N}$
is a scaling factor to balance the multi-head attention.

####  4.2.3 Spatial-Temporal Aggregation

Similar to relevance measuring, we also compute the spatial-temporally
aggregated features $\mathcal{M}^{T}_{i}\in\mathbb{R}^{THW\times\frac{C}{N}}$
within the constrained neighborhood during temporal aggregation. It is
calculated by:

$\mathcal{M}^{T}_{i}=\mathcal{M}^{A}_{i}\mathcal{F}^{S}\langle\mathbf{X}^{a},V_{i}\rangle,~{}\text{when}~{}\mathbf{X}^{a}\in\mathcal{M}^{A}_{i}.$
(5)

####  4.2.4 Soft-Attention

We utilize a soft-attention block to synthesize features from the group of
affinity matrices $\mathcal{M}^{A}_{i}$ and aggregated features
$\mathcal{M}^{T}_{i}$. During the synthesis process, relevant spatial-temporal
patterns should be enhanced while less relevant ones should be suppressed. We
first concatenate a group of affinity matrices $\mathcal{M}^{A}_{i}$ along the
channel dimension to generate $\mathcal{M}^{A}$. The soft-attention map
$\mathcal{M}^{S}$ is computed by:

$\mathcal{M}^{S}\in\mathbb{R}^{THW\times 1}=\texttt{Max}(\mathcal{M}^{A}),$
(6)

where $\mathcal{M}^{A}\in\mathbb{R}^{THW\times T(2k+1)^{2}N}$ and the
$\texttt{Max}(\cdot)$ function computes the channel-wise maximum value. We
then concatenate a group of the spatial-temporally aggregated features
$\mathcal{M}^{T}_{i}$ along the channel dimension to generate
$\mathcal{M}^{T}$.

####  4.2.5 Normalized Self-attention

Finally, our normalized self-attention block, _i.e._ , the function
$\texttt{NS}(\cdot,\cdot,\cdot)$, is defined as:

$\begin{split}Y\in\mathbb{R}^{T\times{H}\times{W}\times
C}&=\texttt{NS}(Q,K,V)\\\
&=(\mathcal{M}^{T}\mathbf{W}_{T})\circledast\mathcal{M}^{S},\end{split}$ (7)

where $\mathbf{W}_{T}$ is the learnable weight and $\circledast$ denotes the
channel-wise Hadamard product.

###  4.3 Global-to-Local Learning

Observation. By establishing the non-local connections for the given features,
the proposed NS block, as in Sec. 4.2, shows the promising potential for
learning short-term spatial-temporal dependencies. However, this mechanism
still struggles in modeling long-term spatial-temporal dependencies due to
limited computational resources, _i.e._ , the network can only process a piece
of frames within a limited time span.

In contrast to our conference version, PNSNet ji2021pnsnet , we propose a
novel global-to-local learning paradigm, which realizes both long-and short-
term spatial-temporal propagation at an arbitrary temporal distance, yielding
a simple but efficient framework, PNS+. Specifically, it appends a spatial-
temporal learning pathway at a global temporal level, naturally introducing
long-term dependencies into the network. We describe this strategy via the
following five steps: a global encoder (Sec. 4.3.1), a local encoder (Sec.
4.3.2), the global spatial-temporal modeling (Sec. 4.3.3), the global-to-local
propagation (Sec. 4.3.4), and the decoder/objectiveness (Sec. 4.3.5).

####  4.3.1 Global Encoder

Our strategy employs the first frame $I_{1}\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times 3}$ as an anchor (_i.e._ , global reference). The dependency
will be calculated between the anchor frame and the sampled consecutive frames
within a sliding window. Following PraNet fan2020pra , we use the same
backbone, Res2Net-50 pami20Res2net , to extract the feature at the conv4_6
layer. To alleviate the computational burden, we adopt an RFB-like
liu2018receptive  module to reduce the channel dimension of extracted feature
and generate the anchor feature $\mathcal{A}^{h}\in\mathbb{R}^{H^{h}\times
W^{h}\times C^{h}}$.

####  4.3.2 Local Encoder

The local encoder takes a piece of consecutive frames
$\mathcal{I}_{\Delta}$$=$$\\{I_{s}\\}_{s=t}^{t+\Delta}\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times 3}~{}(t$$>$$1)$ from a sliding window as input. Similar to
the global encoder, we leverage Res2Net-50 backbone to extract two groups of
short-term features from the conv3_4 and conv4_6 layers and use channel
reduction to generate the low-level
$\\{\mathcal{X}^{l}_{s}\\}_{s=t}^{t+\Delta}$$\in$$\mathbb{R}^{H^{l}\times
W^{l}\times C^{l}}$ and high-level
$\\{\mathcal{X}^{h}_{s}\\}_{s=t}^{t+\Delta}$$\in$$\mathbb{R}^{H^{h}\times
W^{h}\times C^{h}}$ short-term features. We set $H^{l}=\frac{H^{\prime}}{4}$,
$W^{l}=\frac{W^{\prime}}{4}$, $C^{l}=24$, $H^{h}=\frac{H^{\prime}}{8}$,
$W^{h}=\frac{W^{\prime}}{8}$, and $C^{h}=32$ as the default implementation.

####  4.3.3 Global Spatial-Temporal Modeling

As shown in Fig. 5, we leverage the first NS block to model the long-term
relationship at an arbitrary temporal distance, which requires a four-
dimensional temporal feature as input; therefore we have:

$\displaystyle\tilde{\mathcal{X}}^{h}\in\mathbb{R}^{\Delta\times H^{h}\times
W^{h}\times C^{h}}$
$\displaystyle\Leftarrow\\{\mathcal{X}^{h}_{s}\\}_{s=t}^{t+\Delta}\in\mathbb{R}^{H^{h}\times
W^{h}\times C^{h}},$ (8)
$\displaystyle\tilde{\mathcal{A}}^{h}\in\mathbb{R}^{1\times H^{h}\times
W^{h}\times C^{h}}$
$\displaystyle\Leftarrow\mathcal{A}^{h}\in\mathbb{R}^{H^{h}\times W^{h}\times
C^{h}},$

where $\Leftarrow$ denotes reshaping the candidate features into the temporal
form to yield a four-dimensional tensor. Then, as for the first NS block
formulated in Equ.(7), we employ the anchor feature as a query entry (_i.e._ ,
$Q^{g}=\tilde{\mathcal{A}}^{h}$) and the high-level short-term feature as the
key and value entries (_i.e._ , $K^{g}$$=$$\tilde{\mathcal{X}}^{h}$ &
$V^{g}$$=$$\tilde{\mathcal{X}}^{h}$). Intuitively, we aim to build the pixel-
wise similarities between the anchor and high-level short-term features, which
could be viewed as the modeling of global spatial-temporal dependencies. It is
defined as:

$Z^{g}\in\mathbb{R}^{\Delta\times H^{h}\times W^{h}\times
C^{h}}=\texttt{NS}(\tilde{\mathcal{A}}^{h},\tilde{\mathcal{X}}^{h},\tilde{\mathcal{X}}^{h})\oplus\tilde{\mathcal{X}}^{h},$
(9)

where $\oplus$ denotes the element-wise addition of residual operation
he2016deep . This operation provides better convergence stability of interior
gradient propagation within the first NS block, allowing it to easily be
plugged into the pre-trained networks.

####  4.3.4 Global-to-Local Propagation

Furthermore, we desire to propagate the long-term dependency $Z^{g}$ into a
local neighborhood (_i.e._ , frames in a sliding window). Thus, we serve
$Z^{g}$ as the input entries of the second NS block as in Equ.(7), _i.e._ ,
query $Q^{l}=Z^{g}$, key $K^{l}=Z^{g}$, and value $V^{l}=Z^{g}$. We have:

$Z^{l}=\texttt{NS}(Z^{g},Z^{g},Z^{g})\oplus
Z^{g}\oplus\tilde{\mathcal{X}}^{h}.$ (10)

In this way, the introduced two residual connections can maintain the interior
gradient stability (_i.e._ , $\oplus Z^{g}$) and exterior gradient stability
(_i.e._ , $\oplus\tilde{\mathcal{X}}^{h}$) of the second NS block.

####  4.3.5 Decoder and Objectiveness

Finally, we combine the low-level short-term feature $\mathcal{X}^{l}_{s}$
from the local encoder and the spatial-temporal feature $Z^{l}$ from the
second NS block with a two-stage UNet-alike decoder $\mathcal{F}^{D}$. Before
the combination, we recover the feature $Z^{l}$ back to the spatial form,
_i.e._ , $\\{Z^{l}_{s}\\}_{s=t}^{t+\Delta}$. The prediction from the decoder
is computed with:

$\mathcal{P}_{\Delta}=\\{P_{s}\\}_{s=t}^{t+\Delta}=\mathcal{F}^{D}\langle\\{\mathcal{X}^{l}_{s}\\}_{s=t}^{t+\Delta},\\{Z^{l}_{s}\\}_{s=t}^{t+\Delta}\rangle.$
(11)

To this end, given a prediction $P_{s}$ and corresponding ground-truth (GT)
$G_{s}$ at timestamp $s$, we utilize a binary cross-entropy loss for
optimization, which is formulated as:

$\mathcal{L}_{bce}=-\sum[G_{s}\log(P_{s})+(1-G_{s})\log(1-P_{s}))].$ (12)

###  4.4 Implementation Details

####  4.4.1 Datasets

We split $40\%$ SUN-SEG data for training, _i.e._ , SUN-SEG-Train with 112
clips ($19,544$ frames). The rest data are all used for testing, including
SUN-SEG-Easy with 119 clips ($17,070$ frames) and SUN-SEG-Hard with 54 clips
($12,522$ frames) according to difficulty levels in each pathological
category. Specifically, two colonoscopy scenarios (_i.e._ , seen and
unseen333Seen denotes that the samples in the testing dataset are from the
same case in the training set, whereas the unseen indicates that the scenario
do not exist in the training set.) are included in the above two testing
dataset: SUN-SEG-Easy (seen: 33 clips & unseen: 86 clips) and SUN-SEG-Hard
(seen: 17 clips & unseen: 37 clips) for more fine-grained experimental
analyses.

####  4.4.2 Training Details

We train our model using the SUN-SEG-Train dataset on the server platform
equipped with an Intel Xeon (R) CPU E5-2690v4$\times$24 and four NVIDIA Tesla
V100 GPUs with 16 GB memory per one. The ImageNet pre-trained weights of
Res2Net-50 pami20Res2net  are loaded before training, and other newly-added
layers are with Kaiming initialization. We set the batch size to $24$, which
takes about 5 hours to reach convergence after 15 epochs. For each mini-batch
of data, we select the first frame of a video clip as an anchor, randomly
sample five consecutive frames ($\Delta$$=$$5$) from the same clip, and resize
them to 256$\times$448\. The Adam optimizer’s initial learning rate and weight
decay are set to 3e-4 and 1e-4, respectively. We set the number of attention
groups to $N$$=$$4$ as default. For the first NS block, we set kernel size
$k$$=$$3$ and dilation rate $d_{i}$$=$$\\{3,4,3,4\\}$ to capture more long-
term representations with a larger receptive field. For the second one, we set
kernel size $k$$=$$3$ and reduce dilation rate $d_{i}$$=$$\\{1,2,1,2\\}$ to
mainly focus on short-term relationships.

####  4.4.3 Inference Stage

We evaluate PNS+ on SUN-SEG-Easy and SUN-SEG-Hard with both seen and unseen
scenarios. Similar to the training phase, during inference, we select the
first frame as an anchor, sample five video frames ($\Delta$$=$$5$) from a
video clip, and resize them to 256$\times$448\. For the final prediction, we
use the network’s output $\mathcal{P}_{\Delta}$ followed by a Sigmoid
function. The proposed PNS+ achieves a super real-time inference speed of
170fps on a single V100 GPU without any heuristic post-processing techniques,
such as DenseCRF krahenbuhl2011efficient .

##  5 VPS Benchmark

###  5.1 Evaluation Protocols

####  5.1.1 Competitors

We elaborately select eight typical video-based object/polyp segmentation
methods, including COSNet lu2019see , MAT zhou2020matnet , PCSA gu2020pyramid
, 2/3D puyal2020endoscopic , AMD liu2021emergence , DCF zhang2021dynamic ,
FSNet ji2021full , and PNSNet ji2021pnsnet . We also add five image-based
object/polyp segmentation methods to validate the effectiveness of per-frame
prediction ability, including UNet ronneberger2015u , UNet++ zhou2018unetplus
, ACSNet zhang2020adaptive , PraNet fan2020pra , and SANet wei2021shallow .
For a fair comparison, all the competitors utilize the same dataset as our
PNS+ and reach the convergence under their default training settings. Of note,
this paper only focuses on the positive cases in our SUN-SEG dataset and left
negative cases (no polyp) for future work.

####  5.1.2 Evaluation Metrics

To provide deeper insight into the model performance, we use the following six
different metrics for model evaluation between prediction $P_{s}$ and ground-
truth $G_{s}$ at timestamp $s$, including: (a) Dice coefficient (${\rm
Dice}=\frac{2\times|P_{s}\cap G_{s}|}{|P_{s}\cup G_{s}|}$) measures the
similarity between prediction and ground truth mask and penalizes for the
false-positive/-negative predictions. The operators $\cap$, $\cup$, and
$|\cdot|$ denote the intersection, union, and the number of pixels in an area,
respectively. (b) Pixel-wise sensitivity (${\rm Sen}=\frac{|P_{s}\cap
G_{s}|}{|G_{s}|}$) is used to evaluate the true positive prediction of overall
lesion areas. Since the goal of colonoscopy is to screen the polyps with a low
polyp missing rate, people who have the polyps should be highly likely to be
identified. As a result, penalizing the false-negative prediction can be done
by adopting sensitivity which refers to the method’s ability to correctly
detect polyps. (c) Being the harmonic mean of precision and recall that is
weighted by $\beta$, F-measure achanta2009frequency
($F_{\beta}=\frac{(1+\beta^{2})\times{\rm Prc}\times{\rm
Rcl}}{\beta^{2}\times({\rm Prc}+{\rm Rcl})}$) is widely used in measuring
binary masks by combining precision (${\rm Prc}=\frac{|P_{s}\cap
G_{s}|}{|P_{s}|}$) and recall (${\rm Rcl}=\frac{|P_{s}\cap G_{s}|}{|G_{s}|}$)
for more comprehensive evaluation. (d) Suggested by fan2021cognitive ;
cheng2021structure , weighted F-measure margolin2014evaluate
($F_{\beta}^{w}=\frac{(1+\beta^{2})\times{\rm Prc}^{w}\times{\rm
Rcl}^{w}}{\beta^{2}\times({\rm Prc}^{w}+{\rm Rcl}^{w})}$): amend the “Equal-
importance flaw” in Dice and $F_{\beta}$, providing more reliable evaluation
results. Following borji2015salient , we set the factor $\beta^{2}$ of
$F_{\beta}$ and $F_{\beta}^{w}$ as 0.3 and 1, respectively. (e) Different from
the above pixel-wise metrics, structure measure fan2017structure
($\mathcal{S}_{\alpha}=\alpha\times\mathcal{S}_{o}(P_{s},G_{s})+(1-\alpha)\times\mathcal{S}_{r}(P_{s},G_{s})$):
is used to measure the structural similarity at object-aware $\mathcal{S}_{o}$
and region-aware $\mathcal{S}_{r}$, respectively. We use the factor $\alpha$ =
0.5 as default. (f) Fan et al. proposed a human visual perception-based
metric, enhanced-alignment measure fan2018enhanced :
$E_{\phi}=\frac{1}{W\times
H}\sum_{x}^{W}\sum_{y}^{H}\phi(P_{s}(x,y),G_{s}(x,y))$, where $\phi$ is the
enhanced-alignment matrix. $W$ and $H$ are the width and height of ground-
truth $G_{s}$. This metric is inherently suitable for assessing polyps’
heterogeneous location and shape in colonoscopy.

Table 3: Quantitative comparison on two testing sub-datasets with seen colonoscopy scenarios. |  |  SUN-SEG-Easy (Seen)
---
|  SUN-SEG-Hard (Seen)
---
Model | $\mathcal{S}_{\alpha}$ | $E_{\phi}^{mn}$ | $F_{\beta}^{w}$ | Dice | $\mathcal{S}_{\alpha}$ | $E_{\phi}^{mn}$ | $F_{\beta}^{w}$ | Dice
COSNet | 0.845 | 0.836 | 0.727 | 0.804 | 0.785 | 0.772 | 0.626 | 0.725
MAT | 0.879 | 0.861 | 0.731 | 0.833 | 0.840 | 0.821 | 0.652 | 0.776
PCSA | 0.852 | 0.835 | 0.681 | 0.779 | 0.772 | 0.759 | 0.566 | 0.679
2/3D | 0.895 | 0.909 | 0.819 | 0.856 | 0.849 | 0.868 | 0.753 | 0.809
AMD | 0.471 | 0.526 | 0.114 | 0.245 | 0.480 | 0.536 | 0.115 | 0.231
DCF | 0.572 | 0.591 | 0.357 | 0.398 | 0.603 | 0.602 | 0.385 | 0.443
FSNet | 0.890 | 0.895 | 0.818 | 0.873 | 0.848 | 0.859 | 0.755 | 0.828
PNSNet | 0.906 | 0.910 | 0.836 | 0.861 | 0.870 | 0.892 | 0.787 | 0.823
PNS+ | 0.917 | 0.924 | 0.848 | 0.888 | 0.887 | 0.929 | 0.806 | 0.855

As mentioned in Sec. 4.1, the models generate continuous floating predictions;
thus, we need to threshold the floating value into binary ones ranging from 0
to 255. Specifically, we provide the maximum value of Dice and mean value of
$E_{\phi}$, $F_{\beta}$, and Sen under different thresholds for the binary
metrics. Furthermore, we obtain the video-level score by averaging the
evaluated results per image at a video clip. Then, we take the average video-
level scores as the performance on the whole dataset. One-key evaluation
toolbox is available at https://github.com/GewelsJI/VPS/tree/main/eval.

###  5.2 Quantitative Comparison

Based on the protocols mentioned in Sec. 5.1, we conduct a comprehensive VPS
benchmark on two testing sub-datasets (_i.e._ , SUN-SEG-Easy and SUN-SEG-
Hard), which include the following three aspects.

####  5.2.1 Learning Ability

Notably, the image-based models are trained and inferred frame-by-frame. To
better unveil the spatial-temporal learning ability on the colonoscopy videos,
we conduct two groups of experiments to validate video-based competitors’
ability on two seen sub-datasets. For these sub-datasets shown in Table 3, our
PNS+ also outperforms top-1 video-based approaches, _e.g._ , Dice score on
SUN-SEG-Easy (Seen): PNSNet (0.861) vs. PNS+(0.888) and $F^{mn}_{\phi}$ score
on SUN-SEG-Hard (Seen): PNSNet (0.892) vs. PNS+(0.929). The above results
suggest that our model has a strong learning ability to accurately segment
polyps.

####  5.2.2 Generalization Capability

To validate the model’s generalizability, we conduct the experiments on two
testing sub-datasets with unseen colonoscopy scenarios. As shown in Table 4,
we present the performance comparison with the other latest image- and video-
based competitors in six metrics. It shows that our PNS+ achieves significant
improvements by a large margin in comparison with top image-and video-based
approaches, _e.g._ , Dice score on SUN-SEG-Easy (Unseen): ACSNet (0.713) vs.
2/3D (0.722) vs. PNS+ (0.756) and $F^{w}_{\beta}$ score on SUN-SEG-Hard
(Unseen): ACSNet (0.636) vs. 2/3D (0.634) vs. PNS+ (0.653). Interestingly, we
observe that PNSNet drops dramatically on two unseen datasets, which is a side
show of better generalizability attributed to our newly-proposed global-to-
local learning strategy, especially on a clip with a larger time span.

Table 4: Quantitative comparison of two testing sub-datasets with unseen colonoscopy scenarios. ‘R/T’ means to retrain the private model using the code provided by the author. The best values are highlighted in bold. |  |  |  |  |  SUN-SEG-Easy (Unseen)
---
|  SUN-SEG-Hard (Unseen)
---
| Model | Publish | Code | $\mathcal{S}_{\alpha}$ | $E_{\phi}^{mn}$ | $F_{\beta}^{w}$ | $F_{\beta}^{mn}$ | Dice | Sen | $\mathcal{S}_{\alpha}$ | $E_{\phi}^{mn}$ | $F_{\beta}^{w}$ | $F_{\beta}^{mn}$ | Dice | Sen
IMAGE |  UNet ronneberger2015u  |  MICCAI15 | https://github.com/4uiiurz1/pytorch-nested-unet | 0.669 | 0.677 | 0.459 | 0.528 | 0.530 | 0.420 | 0.670 | 0.679 | 0.457 | 0.527 | 0.542 | 0.429
UNet++ zhou2018unetplus  |  TMI18 | https://github.com/MrGiovanni/UNetPlusPlus | 0.684 | 0.687 | 0.491 | 0.553 | 0.559 | 0.457 | 0.685 | 0.697 | 0.480 | 0.544 | 0.554 | 0.467
ACSNet zhang2020adaptive  |  MICCAI20 | https://github.com/ReaFly/ACSNet | 0.782 | 0.779 | 0.642 | 0.688 | 0.713 | 0.601 | 0.783 | 0.787 | 0.636 | 0.684 | 0.708 | 0.618
PraNet fan2020pra  |  MICCAI20 | https://github.com/DengPingFan/PraNet | 0.733 | 0.753 | 0.572 | 0.632 | 0.621 | 0.524 | 0.717 | 0.735 | 0.544 | 0.607 | 0.598 | 0.512
SANet wei2021shallow  |  MICCAI21 | https://github.com/weijun88/SANet | 0.720 | 0.745 | 0.566 | 0.634 | 0.649 | 0.521 | 0.706 | 0.743 | 0.526 | 0.580 | 0.598 | 0.505
VIDEO |  COSNet lu2019see  |  TPAMI19 | https://github.com/carrierlxk/COSNet | 0.654 | 0.600 | 0.431 | 0.496 | 0.596 | 0.359 | 0.670 | 0.627 | 0.443 | 0.506 | 0.606 | 0.380
MAT zhou2020matnet  |  TIP20 | https://github.com/tfzhou/MATNet | 0.770 | 0.737 | 0.575 | 0.641 | 0.710 | 0.542 | 0.785 | 0.755 | 0.578 | 0.645 | 0.712 | 0.579
PCSA gu2020pyramid  |  AAAI20 | https://github.com/guyuchao/PyramidCSA | 0.680 | 0.660 | 0.451 | 0.519 | 0.592 | 0.398 | 0.682 | 0.660 | 0.442 | 0.510 | 0.584 | 0.415
2/3D puyal2020endoscopic  |  MICCAI20 | R/T | 0.786 | 0.777 | 0.652 | 0.708 | 0.722 | 0.603 | 0.786 | 0.775 | 0.634 | 0.688 | 0.706 | 0.607
AMD liu2021emergence  |  NeurIPS21 | https://github.com/rt219/the-emergence-of-objectness | 0.474 | 0.533 | 0.133 | 0.146 | 0.266 | 0.222 | 0.472 | 0.527 | 0.128 | 0.141 | 0.252 | 0.213
DCF zhang2021dynamic  |  ICCV21 | https://github.com/Roudgers/DCFNet | 0.523 | 0.514 | 0.270 | 0.312 | 0.325 | 0.340 | 0.514 | 0.522 | 0.263 | 0.303 | 0.317 | 0.364
FSNet ji2021full  |  ICCV21 | https://github.com/GewelsJI/FSNet | 0.725 | 0.695 | 0.551 | 0.630 | 0.702 | 0.493 | 0.724 | 0.694 | 0.541 | 0.611 | 0.699 | 0.491
PNSNet ji2021pnsnet  |  MICCAI21 | https://github.com/GewelsJI/PNS-Net | 0.767 | 0.744 | 0.616 | 0.664 | 0.676 | 0.574 | 0.767 | 0.755 | 0.609 | 0.656 | 0.675 | 0.579
| PNS+ | OURS22 | https://github.com/GewelsJI/VPS | 0.806 | 0.798 | 0.676 | 0.730 | 0.756 | 0.630 | 0.797 | 0.793 | 0.653 | 0.709 | 0.737 | 0.623
Table 5: Visual attributes-based performance on SUN-SEG-Easy/-Hard (Unseen) in terms of structure measure ($\mathcal{S}_{\alpha}$) score. |  |  SUN-SEG-Easy (Unseen)
---
|  SUN-SEG-Hard (Unseen)
---
| SI | IB | HO | GH | FM | SO | LO | OC | OV | SV | SI | IB | HO | GH | FM | SO | LO | OC | OV | SV
UNet | 0.675 | 0.548 | 0.768 | 0.715 | 0.633 | 0.593 | 0.648 | 0.670 | 0.643 | 0.620 | 0.618 | 0.619 | 0.663 | 0.676 | 0.713 | 0.689 | 0.633 | 0.658 | 0.659 | 0.658
UNet++ | 0.701 | 0.542 | 0.782 | 0.739 | 0.647 | 0.591 | 0.678 | 0.683 | 0.665 | 0.617 | 0.654 | 0.604 | 0.665 | 0.696 | 0.714 | 0.681 | 0.660 | 0.676 | 0.677 | 0.678
ACSNet | 0.789 | 0.612 | 0.896 | 0.820 | 0.704 | 0.663 | 0.787 | 0.770 | 0.759 | 0.705 | 0.770 | 0.681 | 0.828 | 0.795 | 0.817 | 0.738 | 0.810 | 0.828 | 0.806 | 0.759
PraNet | 0.745 | 0.585 | 0.821 | 0.772 | 0.673 | 0.611 | 0.722 | 0.722 | 0.703 | 0.653 | 0.673 | 0.635 | 0.725 | 0.720 | 0.755 | 0.691 | 0.666 | 0.714 | 0.708 | 0.703
SANet | 0.724 | 0.582 | 0.854 | 0.760 | 0.676 | 0.615 | 0.703 | 0.701 | 0.711 | 0.680 | 0.658 | 0.565 | 0.738 | 0.709 | 0.760 | 0.692 | 0.733 | 0.729 | 0.727 | 0.693
COSNet | 0.663 | 0.531 | 0.786 | 0.684 | 0.610 | 0.549 | 0.637 | 0.648 | 0.613 | 0.617 | 0.641 | 0.593 | 0.727 | 0.668 | 0.690 | 0.637 | 0.694 | 0.707 | 0.666 | 0.625
MAT | 0.772 | 0.664 | 0.873 | 0.789 | 0.706 | 0.691 | 0.755 | 0.738 | 0.746 | 0.715 | 0.772 | 0.701 | 0.801 | 0.776 | 0.782 | 0.780 | 0.791 | 0.795 | 0.789 | 0.750
PCSA | 0.676 | 0.563 | 0.759 | 0.708 | 0.628 | 0.610 | 0.634 | 0.662 | 0.656 | 0.616 | 0.656 | 0.591 | 0.692 | 0.683 | 0.706 | 0.671 | 0.612 | 0.677 | 0.665 | 0.663
2/3D | 0.809 | 0.625 | 0.899 | 0.835 | 0.728 | 0.667 | 0.820 | 0.783 | 0.778 | 0.719 | 0.768 | 0.662 | 0.865 | 0.784 | 0.797 | 0.737 | 0.853 | 0.827 | 0.808 | 0.765
AMD | 0.476 | 0.461 | 0.471 | 0.481 | 0.484 | 0.466 | 0.447 | 0.467 | 0.442 | 0.498 | 0.471 | 0.468 | 0.447 | 0.473 | 0.468 | 0.469 | 0.453 | 0.487 | 0.462 | 0.481
DCF | 0.465 | 0.485 | 0.479 | 0.505 | 0.541 | 0.495 | 0.362 | 0.484 | 0.492 | 0.495 | 0.441 | 0.508 | 0.422 | 0.498 | 0.587 | 0.556 | 0.351 | 0.470 | 0.494 | 0.540
FSNet | 0.719 | 0.603 | 0.810 | 0.752 | 0.694 | 0.632 | 0.686 | 0.711 | 0.691 | 0.665 | 0.662 | 0.648 | 0.743 | 0.713 | 0.774 | 0.723 | 0.701 | 0.728 | 0.728 | 0.694
PNSNet | 0.789 | 0.592 | 0.871 | 0.820 | 0.723 | 0.619 | 0.768 | 0.749 | 0.751 | 0.705 | 0.746 | 0.631 | 0.803 | 0.780 | 0.778 | 0.743 | 0.805 | 0.790 | 0.794 | 0.758
PNS+ | 0.819 | 0.667 | 0.883 | 0.844 | 0.738 | 0.690 | 0.796 | 0.782 | 0.798 | 0.734 | 0.770 | 0.703 | 0.817 | 0.801 | 0.823 | 0.793 | 0.792 | 0.808 | 0.807 | 0.795

####  5.2.3 Attribute-based Performance

Finally, we analyze the visual attribute-based comparison presented in Table
2. In terms of $\mathcal{S}_{\alpha}$ score, Table 5 unveils that our PNS+
consistently outperforms other rivals on four attributes (_i.e._ , IB, GH, FM,
and SV). More specifically, as shown in Table 5, most methods can not address
the VPS tasks with IB attribute since the colon polyps always have fuzzy
boundaries. In contrast, PNS+ achieves the best score
($\mathcal{S}_{\alpha}=0.667$) on this challenging IB attribute of SUN-SEG-
Easy (Unseen). This discovery is also consistent with the results shown in
Fig. 6. Similarly, the SO attributes also present the lower scores (_e.g._ ,
SUN-SEG-Easy (Unseen): $\mathcal{S}_{\alpha}$$=$$0.667$), which indicates
these two attributes are the most challenging issues in colonoscopy. On the
contrary, HO and LO attribute consistently sustain higher scores than other
attributes, making polyp easier to detect. This phenomenon meets our
expectations since the less distribution bias for these relatively easy
scenarios. We refer the readers to Sec. 5.5 for a more visualized analysis of
challenging cases.

###  5.3 Qualitative Comparison

As shown in Fig. 6, we present visual results on three video clips of four
typical models (_i.e._ , PNSNet, 2/3D, MAT, ACSNet) and our PNS+. In the last
four rows, the competitors fail to generate complete segmentation results for
the polyps that share the same camouflaged texture with the background. In
contrast, in the $3^{rd}$ row, our model can accurately locate and segment
polyps in a challenging situation, _i.e._ , polyps with different sizes and
homogeneous textures.

###  5.4 Ablation Studies

To validate the effectiveness of our core designs, we conduct extensive
ablation studies and summarize the results in Table 6.

####  5.4.1 Contribution of Base Network

We initialize an UNet-like variant #01 via leveraging the Res2Net-50
pami20Res2net  backbone, which can be viewed as an image-based approach to
generate per frame predictions. We observe that #OUR significantly improves
the performance ($\mathcal{S}_{\alpha}$: +7.7%) on SUN-SEG-Easy (Unseen).

Table 6: Ablation studies for the core designs of the proposed PNS+. See Sec. 5.4 for the detailed analyses. |  | VARIANTS
---
|  SUN-SEG-Easy (Unseen)
---
|  SUN-SEG-Hard (Unseen)
---
No. | Base | $N$ | Soft | Norm | Strategy | $\mathcal{S}_{\alpha}$ | $E_{\phi}^{mn}$ | $F_{\beta}^{w}$ | Dice | $\mathcal{S}_{\alpha}$ | $E_{\phi}^{mn}$ | $F_{\beta}^{w}$ | Dice
#01 | ✓ | - | - | - | - | 0.729 | 0.718 | 0.571 | 0.616 | 0.726 | 0.720 | 0.559 | 0.603
#02 | ✓ | 1 | ✓ | ✓ | L | 0.782 | 0.766 | 0.631 | 0.722 | 0.783 | 0.775 | 0.629 | 0.715
#03 | ✓ | 2 | ✓ | ✓ | L | 0.773 | 0.760 | 0.625 | 0.720 | 0.785 | 0.784 | 0.631 | 0.719
#04 | ✓ | 4 | ✓ | ✓ | L | 0.786 | 0.777 | 0.651 | 0.741 | 0.792 | 0.789 | 0.649 | 0.735
#05 | ✓ | 8 | ✓ | ✓ | L | 0.774 | 0.762 | 0.627 | 0.724 | 0.775 | 0.774 | 0.619 | 0.708
#06 | ✓ | 4 | - | ✓ | L | 0.782 | 0.775 | 0.639 | 0.722 | 0.785 | 0.786 | 0.637 | 0.715
#07 | ✓ | 4 | ✓ | - | L | 0.755 | 0.752 | 0.587 | 0.705 | 0.754 | 0.751 | 0.579 | 0.694
#08 | ✓ | 4 | ✓ | ✓ |  L$\rightarrow$L | 0.748 | 0.717 | 0.577 | 0.705 | 0.760 | 0.741 | 0.587 | 0.693
#09 | ✓ | 4 | ✓ | ✓ |  L$\rightarrow$G | 0.788 | 0.780 | 0.645 | 0.741 | 0.776 | 0.768 | 0.618 | 0.715
#10 | ✓ | 4 | ✓ | ✓ |  G$\rightarrow$G | 0.778 | 0.763 | 0.627 | 0.726 | 0.767 | 0.753 | 0.599 | 0.694
#OUR | ✓ | 4 | ✓ | ✓ | G$\rightarrow$L | 0.806 | 0.798 | 0.676 | 0.756 | 0.797 | 0.793 | 0.653 | 0.737
Figure 6: Qualitative visualization of the proposed PNS+ and four
representative competitors on three sequences (from left to right: case14_3,
case30, and case3_2). The red boxes indicates the wrong or missing
predictions. We refer the readers to the project page for completed dynamic
comparison.

####  5.4.2 Contribution of Channel Split

To discover the best setting for the channel split rule as in Equ.(1), we
instantiate four variants with four different channel split numbers: #02
($N$$=$$1$), #03 ($N$$=$$2$), #04 ($N$$=$$4$), and #05 ($N$$=$$8$). These
results show that small (#02 & #03) and large (#05) channel split numbers may
harm the channel-level information by collapsing the knowledge in a different
channel. In contrast, we adopt the moderate scale (#04: $N$$=$$4$) with the
best performance on SUN-SEG-Hard (Unseen) (_e.g._ , Dice: 2.7%$\uparrow$) when
compared to variant #05. Such a trade-off scale would exert our model focusing
on the polyp-related attention while suppressing the irrelevant information.

####  5.4.3 Contribution of Soft-attention

We further ablate soft-attention and observe that #04 with the soft-attention
block is generally better than #06 without it on SUN-SEG-Easy (Unseen):
1.9%$\uparrow$ in terms of Dice score. Such improvement suggests that
introducing the soft-attention operation to synthesize the relationship
between aggregation feature and affinity matrix is necessary for increasing
performance.

####  5.4.4 Effectiveness of Normalization

We also study the improvement of the normalization operation by comparing #04
with #07. We observe that #04 generally outperforms #07 on SUN-SEG-Hard
(Unseen) (_e.g._ , Dice: 4.1%$\uparrow$). It shows that the layer
normalization along the temporal dimension could alleviate the internal
covariate shift problem by fixing the distribution of query entries in the
attention mechanism.

(a) Anchor frame $I_{1}$(b) Current frame $I_{s}$(c) Ground truth $G_{s}$(d)
Anchorfeature $\mathcal{A}^{h}$(e) Spatial-temporalfeature $Z^{l}$(f)
Prediction $P_{s}$

Figure 7: Feature visualization of key dataflows. The red arrow denotes using
the anchor feature $\mathcal{A}^{h}$ to guide the representation of spatial-
temporal frame $Z^{l}$. More details refer to Sec. 5.4.5.

####  5.4.5 Different Learning Strategies

Finally, we examine the effectiveness of the proposed learning strategy, as
proposed in Sec. 4.3, by deriving three variants, including #08
(L$\rightarrow$L: local-to-local), #09 (L$\rightarrow$G: local-to-global), #10
(G$\rightarrow$G: global-to-global), and #OUR (G$\rightarrow$L: global-to-
local). For example, variant #09 combines local spatial-temporal cues and
introduces global ones, termed a local-to-global (L$\rightarrow$G) strategy.
#08 will dramatically decrease on SUN-SEG-Easy (Unseen)
($\mathcal{S}_{\alpha}$: 5.8%$\downarrow$) when focusing on the local cues due
to a lack of global context. On the other hand, if only focusing on the global
information, the performance of variant #10 will drop on SUN-SEG-Hard
(Unseen), _e.g._ , $F_{\beta}^{w}$: 5.4%$\downarrow$. In contrast, #OUR with
the global-to-local strategy outperforms variant #09 on SUN-SEG-Hard (Unseen),
_e.g._ , $F_{\beta}^{w}$: 3.5%$\uparrow$, since propagating long-term cues
into short-term neighbors.

We further validate the effectiveness of the global-to-local learning strategy
via visualizing the key dataflows. As shown in Fig. 7, the first and second
columns present the anchor feature $\mathcal{A}^{h}$ extracted from the global
encoder and the spatial-temporal feature $Z^{l}$ from the second NS block,
respectively. Note that the current frame $I_{s}$ is randomly selected from
consecutive frames $I_{\Delta}$. It shows that our PNS+ can propagate the
long-term dependency with the assistance of the anchor frame $I_{1}$, though
the current frame $I_{s}$ is hard to recognize due to indefinable boundaries
(_i.e._ , IB attribute). Of note, as in the rightmost column of Fig. 6, the
PNSNet fails to locate the polyp since it does not use a global-to-local
learning strategy. Compared to it, our PNS+ successfully detects the polyp by
exploiting the global reference of the anchor frame.

Figure 8: Challenging samples were taken from ten visual attributes. More
analyses can be referred to Sec. 5.5.

###  5.5 Issues and Challenges

This section discusses some common issues within challenging attributes, whose
visualization results are presented in Fig. 8. Of note, VPS is a newly-
emerging and challenging track in medical imaging, and its overall accuracy is
not high enough. We observe that existing cutting-edge models (_i.e._ , ACSNet
and 2/3D) and our baseline model (PNS+) still lack sufficient robustness in
particular cases in LO, HO, SI, GH, and SV attributes. As for the HO ($3^{rd}$
column) and LO ($8^{th}$ column) attributes, three models fail to capture the
whole polyp due to significant appearance changes. Besides, the false-
positive/-negative predictions (marked with red boxes) on the surgical
instrument ($1^{st}$ column) and the optical flares ($4^{th}$ column) indicate
that these models could not learn semantics without perceiving the accurate
polyp-related representation in such a hard case. Moreover, the
misidentifications for the SV attribute (last column) are caused by the
insufficient diversity of polyp sizes in the training set. The aforementioned
drawbacks inspire us to explore more robust learning paradigms to improve the
accuracy of VPS.

We also observe that three models consistently fail to locate lesion regions
that share a similar color with the intestinal wall or are too small to be
detected. Thus, there is a large room for improving the detection ability in
IB and SO attributes via camouflaged pattern discovery techniques
fan2021concealed ; ji2022gradient . Last but not least, lacking temporal-wise
understanding will lead to the false prediction in the FM, OV, and OC
attributes. Taking OV and OC, for example, exploiting temporal cues more
thoroughly should mitigate the performance degradation results from the
occlusion of the intestinal wall or the image boundary since the occlusion is
not continuous in the entire video clip. To sum up, these challenging cases
are the common difficulties other methods face and cause a severe performance
degradation that deserves further exploration.

##  6 Potential Directions

This section highlights several potential trends for promoting colonoscopy
research in the deep era.

  * •

High-precision Diagnosis. As shown in Table 4, we observe that the leading
approaches are still unsatisfactory in our SUN-SEG-Hard (_e.g._ , sensitivity
score $<$ 0.63). We argue that the high-precision VPS algorithm would steer
clinical medicine in boosting auxiliary diagnostic technologies.

  * •

Data-insufficient Learning. It is promising to explore efficient learning
strategies guo2021semantic ; senkyire2021supervised  under limited conditions
in specific clinical applications, such as weakly-/un-/self-supervised
learning and knowledge distillation.

  * •

Privacy-preserving AI. Intelligent VPS systems must safeguard data through the
entire life cycle from training to production and governance, which fuels
fundamental techniques like federal learning.

  * •

Trustworthy AI. How AI-guided decisions are made and what determining factors
are involved play a crucial role in understanding the insights of deep
networks. In other words, the VPS model should be causal, transparent,
explainable, and interactive, which inspires more trusted developments, such
as zou2022tbrats .

The above possible directions listed are still far from being solved for the
VPS. Fortunately, several famous works can be served as references, providing
it a potential basis to be transferred to our community.

##  7 Conclusion

This paper presents the first comprehensive study on video polyp segmentation
(VPS) from a deep learning perspective. We first introduce a large-scale VPS
dataset SUN-SEG via extending the famous SUN-database with diversified
annotations, _i.e._ , attribute, object mask, boundary, scribble, and polygon.
We then design a simple but efficient baseline, dubbed PNS+, to segment colon
polyps from the colonoscopy video. Based on the normalized self-attention
block, PNS+ fully exploits long-term and short-term spatial-temporal cues via
a novel global-to-local learning strategy. We further contribute the first
comprehensive benchmark containing 13 cutting-edge polyp/object segmentation
approaches. Extensive results show that PNS+ achieves the best performance
against all these competitors. We conclude by outlining several potential
directions for future colonoscopy-related research in the deep learning era.
We hope this work will spur advancements in other closely related medical
video analyses.

## Acknowledgments

The authors would like to thank the anonymous reviewers and editor for their
helpful comments on this manuscript. Besides, we thank Huazhu Fu for his
insightful feedback.

## Conflicts of Interests

The authors declared that they have no conflicts of interest in this work. We
declare that we do not have any commercial or associative interest that
represents a conflict of interest in connection with the work submitted.

## References

  * (1) J. Bernal, J. Sánchez, and F. Vilarino, “Towards automatic polyp detection with a polyp appearance model,” _Pattern Recognition_ , vol. 45, no. 9, pp. 3166–3182, 2012, DOI: 10.1016/j.patcog.2012.03.002.
  * (2) J. G.-B. Puyal, K. K. Bhatia, P. Brandao, O. F. Ahmad, D. Toth, R. Kader, L. Lovat, P. Mountney, and D. Stoyanov, “Endoscopic polyp segmentation using a hybrid 2d/3d cnn,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Lima, Peru: Springer, 2020, pp. 295–305, DOI: 10.1007/978-3-030-59725-2_29.
  * (3) M. Misawa, S.-e. Kudo, Y. Mori, K. Hotta, K. Ohtsuka, T. Matsuda, S. Saito, T. Kudo, T. Baba, F. Ishida _et al._ , “Development of a computer-aided detection system for colonoscopy and a publicly accessible large colonoscopy video database (with video),” _Gastrointestinal endoscopy_ , vol. 93, no. 4, pp. 960–967, 2021, DOI: 10.1016/j.gie.2020.07.060.
  * (4) G.-P. Ji, Y.-C. Chou, D.-P. Fan, G. Chen, H. Fu, D. Jha, and L. Shao, “Progressively normalized self-attention network for video polyp segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Strasbourg, France: Springer, 2021, pp. 142–152, DOI: 10.1007/978-3-030-87193-2_14.
  * (5) J. Silva, A. Histace, O. Romain, X. Dray, and B. Granado, “Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer,” _International journal of computer assisted radiology and surgery_ , vol. 9, no. 2, pp. 283–293, 2014, DOI: 10.1007/s11548-013-0926-3.
  * (6) J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, D. Gil, C. Rodríguez, and F. Vilariño, “Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians,” _Computerized Medical Imaging and Graphics_ , vol. 43, pp. 99–111, 2015, DOI: 10.1016/j.compmedimag.2015.02.007.
  * (7) P. Mesejo, D. Pizarro, A. Abergel, O. Rouquette, S. Beorchia, L. Poincloux, and A. Bartoli, “Computer-aided classification of gastrointestinal lesions in regular colonoscopy,” _Transactions on Medical Imaging_ , vol. 35, no. 9, pp. 2051–2063, 2016, DOI: 10.1109/TMI.2016.2547947.
  * (8) N. Tajbakhsh, S. R. Gurudu, and J. Liang, “Automated polyp detection in colonoscopy videos using shape and context information,” _Transactions on Medical Imaging_ , vol. 35, no. 2, pp. 630–644, 2016, DOI: 10.1109/TMI.2015.2487997.
  * (9) “Gastrointestinal Image ANAlysis (GIANA) Challenge,” https://endovissub2017-giana.grand-challenge.org/home/.
  * (10) D. Vázquez, J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, A. M. López, A. Romero, M. Drozdzal, and A. Courville, “A benchmark for endoluminal scene segmentation of colonoscopy images,” _Journal of Healthcare Engineering_ , vol. 2017, p. 4037190, 2017, DOI: 10.1155/2017/4037190.
  * (11) A. Koulaouzidis, D. K. Iakovidis, D. E. Yung, E. Rondonotti, U. Kopylov, J. N. Plevris, E. Toth, A. Eliakim, G. W. Johansson, W. Marlicz _et al._ , “Kid project: an internet-based digital video atlas of capsule endoscopy for research purposes,” _Endoscopy International Open_ , vol. 5, no. 6, p. E477, 2017, DOI: 10.1055/s-0043-105488.
  * (12) D. K. Iakovidis, S. V. Georgakopoulos, M. Vasilakakis, A. Koulaouzidis, and V. P. Plagianakos, “Detecting and locating gastrointestinal anomalies using deep learning and iterative cluster unification,” _Transactions on Medical Imaging_ , vol. 37, no. 10, pp. 2196–2210, 2018, DOI: 10.1109/TMI.2018.2837002.
  * (13) K. Pogorelov, K. R. Randel, C. Griwodz, S. L. Eskeland, T. de Lange, D. Johansen, C. Spampinato, D.-T. Dang-Nguyen, M. Lux, P. T. Schmidt _et al._ , “Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection,” in _Multimedia Systems Conference_. Taipei, Taiwan: ACM, 2017, pp. 164–169, DOI: 10.1145/3083187.3083212.
  * (14) S. Ali, N. Ghatwary, B. Braden, D. Lamarque, A. Bailey, S. Realdon, R. Cannizzaro, J. Rittscher, C. Daul, and J. East, “Endoscopy disease detection challenge 2020,” [Online], 2020, available: https://arxiv.org/abs/2003.03376.
  * (15) H. Borgli, V. Thambawita, P. H. Smedsrud, S. Hicks, D. Jha, S. L. Eskeland, K. R. Randel, K. Pogorelov, M. Lux, D. T. D. Nguyen _et al._ , “Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy,” _Scientific Data_ , vol. 7, no. 1, pp. 1–14, 2020, DOI: 10.1038/s41597-020-00622-y.
  * (16) D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. de Lange, D. Johansen, and H. D. Johansen, “Kvasir-seg: A segmented polyp dataset,” in _International Conference on Multimedia Modeling_. Daejeon, Korea: Springer, 2020, pp. 451–462, DOI: 10.1007/978-3-030-37734-2_37.
  * (17) L. F. Sánchez-Peralta, J. B. Pagador, A. Picón, Á. J. Calderón, F. Polo, N. Andraka, R. Bilbao, B. Glover, C. L. Saratxaga, and F. M. Sánchez-Margallo, “Piccolo white-light and narrow-band imaging colonoscopic dataset: A performance comparative of models and datasets,” _Applied Sciences_ , vol. 10, no. 23, p. 8501, 2020, DOI: 10.3390/app10238501.
  * (18) P. H. Smedsrud, V. Thambawita, S. A. Hicks, H. Gjestang, O. O. Nedrejord, E. Næss, H. Borgli, D. Jha, T. J. D. Berstad, S. L. Eskeland _et al._ , “Kvasir-capsule, a video capsule endoscopy dataset,” _Scientific Data_ , vol. 8, no. 1, pp. 1–10, 2021, DOI: 10.1038/s41597-021-00920-z.
  * (19) W. Wang, J. Tian, C. Zhang, Y. Luo, X. Wang, and J. Li, “An improved deep learning approach and its applications on colonic polyp images detection,” _BMC Medical Imaging_ , vol. 20, no. 1, pp. 1–14, 2020, DOI: 10.1186/s12880-020-00482-3.
  * (20) Y. Ma, X. Chen, K. Cheng, Y. Li, and B. Sun, “Ldpolypvideo benchmark: A large-scale colonoscopy video dataset of diverse polyps,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Strasbourg, France: Springer, 2021, pp. 387–396, DOI: 10.1007/978-3-030-87240-3_37.
  * (21) K. Li, M. I. Fathan, K. Patel, T. Zhang, C. Zhong, A. Bansal, A. Rastogi, J. S. Wang, and G. Wang, “Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations,” _Plos one_ , vol. 16, no. 8, p. e0255809, 2021, DOI: 10.1371/journal.pone.0255809.
  * (22) S. Ali, D. Jha, N. Ghatwary, S. Realdon, R. Cannizzaro, O. E. Salem, D. Lamarque, C. Daul, K. V. Anonsen, M. A. Riegler _et al._ , “Polypgen: A multi-center polyp detection and segmentation dataset for generalisability assessment,” [Online], 2021, available: https://arxiv.org/abs/2106.04463.
  * (23) B. V. Dhandra, R. Hegadi, M. Hangarge, and V. S. Malemath, “Analysis of abnormality in endoscopic images using combined hsi color space and watershed segmentation,” in _International Conference on Pattern Recognition_. Hong Kong, China: IEEE, 2006, pp. 695–698, DOI: 10.1109/ICPR.2006.268.
  * (24) A. V. Mamonov, I. N. Figueiredo, P. N. Figueiredo, and Y.-H. R. Tsai, “Automated polyp detection in colon capsule endoscopy,” _Transactions on Medical Imaging_ , vol. 33, no. 7, pp. 1488–1502, 2014, DOI: 10.1109/TMI.2014.2314959.
  * (25) O. H. Maghsoudi, “Superpixel based segmentation and classification of polyps in wireless capsule endoscopy,” in _Signal Processing in Medicine and Biology Symposium_. Philadelphia, PA, USA: IEEE, 2017, pp. 1–4, DOI: 10.1109/SPMB.2017.8257027.
  * (26) L. Yu, H. Chen, Q. Dou, J. Qin, and P. A. Heng, “Integrating online and offline three-dimensional deep learning for automated polyp detection in colonoscopy videos,” _Journal of Biomedical and Health Informatics_ , vol. 21, no. 1, pp. 65–75, 2016, DOI: 10.1109/JBHI.2016.2637004.
  * (27) W. Tavanapong, J. Oh, M. Riegler, M. I. Khaleel, B. Mitta, and P. C. De Groen, “Artificial intelligence for colonoscopy: Past, present, and future,” _Journal of Biomedical and Health Informatics_ , pp. 1–1, 2022, DOI: 10.1109/JBHI.2022.3160098.
  * (28) H. Gammulle, S. Denman, S. Sridharan, and C. Fookes, “Two-stream deep feature modelling for automated video endoscopy data analysis,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Lima, Peru: Springer, 2020, pp. 742–751, DOI: 10.1007/978-3-030-59716-0_71.
  * (29) G. Carneiro, L. Z. C. T. Pu, R. Singh, and A. Burt, “Deep learning uncertainty and confidence calibration for the five-class polyp classification from colonoscopy,” _Medical image analysis_ , vol. 62, p. 101653, 2020, DOI: 10.1016/j.media.2020.101653.
  * (30) R. Zhang, Y. Zheng, C. C. Poon, D. Shen, and J. Y. Lau, “Polyp detection during colonoscopy using a regression-based convolutional neural network with a tracker,” _Pattern Recognition_ , vol. 83, pp. 209–219, 2018, DOI: 10.1016/j.patcog.2018.05.026.
  * (31) L. Wu, Z. Hu, Y. Ji, P. Luo, and S. Zhang, “Multi-frame collaboration for effective endoscopic video polyp detection via spatial-temporal feature transformation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Strasbourg, France: Springer, 2021, pp. 302–312, DOI: 10.1007/978-3-030-87240-3_29.
  * (32) P. Brandao, E. Mazomenos, G. Ciuti, R. Caliò, F. Bianchi, A. Menciassi, P. Dario, A. Koulaouzidis, A. Arezzo, and D. Stoyanov, “Fully convolutional neural networks for polyp segmentation in colonoscopy,” in _Medical Imaging 2017: Computer-Aided Diagnosis_. Orlando, FL, USA: SPIE, 2017, pp. 101–107, DOI: 10.1117/12.2254361.
  * (33) M. Akbari, M. Mohrekesh, E. Nasr-Esfahani, S. R. Soroushmehr, N. Karimi, S. Samavi, and K. Najarian, “Polyp segmentation in colonoscopy images using fully convolutional network,” in _Engineering in Medicine and Biology Society_. Honolulu, HI, USA: IEEE, 2018, pp. 69–72, DOI: 10.1109/EMBC.2018.8512197.
  * (34) O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Munich, Germany: Springer, 2015, pp. 234–241, DOI: 10.1007/978-3-319-24574-4_28.
  * (35) Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: Redesigning skip connections to exploit multiscale features in image segmentation,” _Transactions on Medical Imaging_ , vol. 39, no. 6, pp. 1856–1867, 2019, DOI: 10.1109/TMI.2019.2959609.
  * (36) D. Jha, P. H. Smedsrud, M. A. Riegler, D. Johansen, T. De Lange, P. Halvorsen, and H. D. Johansen, “Resunet++: An advanced architecture for medical image segmentation,” in _International Symposium on Multimedia_. San Diego, CA, USA: IEEE, 2019, pp. 225–2255, DOI: 10.1109/ISM46123.2019.00049.
  * (37) J. Zhong, W. Wang, H. Wu, Z. Wen, and J. Qin, “Polypseg: An efficient context-aware network for polyp segmentation from colonoscopy videos,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Lima, Peru: Springer, 2020, pp. 285–294, DOI: 10.1007/978-3-030-59725-2_28.
  * (38) R. Zhang, G. Li, Z. Li, S. Cui, D. Qian, and Y. Yu, “Adaptive context selection for polyp segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Lima, Peru: Springer, 2020, pp. 253–262, DOI: 10.1007/978-3-030-59725-2_25.
  * (39) D. Jha, S. Ali, N. K. Tomar, H. D. Johansen, D. Johansen, J. Rittscher, M. A. Riegler, and P. Halvorsen, “Real-time polyp detection, localization and segmentation in colonoscopy using deep learning,” _IEEE Access_ , vol. 9, pp. 40 496–40 510, 2021, DOI: 10.1109/ACCESS.2021.3063716.
  * (40) H. Wu, J. Zhong, W. Wang, Z. Wen, and J. Qin, “Precise yet efficient semantic calibration and refinement in convnets for real-time polyp segmentation from colonoscopy videos,” in _AAAI Conference on Artificial Intelligence_. [Online]: AAAI Press, 2021, pp. 2916–2924.
  * (41) J. Wei, Y. Hu, R. Zhang, Z. Li, S. K. Zhou, and S. Cui, “Shallow attention network for polyp segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Strasbourg, France: Springer, 2021, pp. 699–708, DOI: 10.1007/978-3-030-87193-2_66.
  * (42) X. Zhao, L. Zhang, and H. Lu, “Automatic polyp segmentation via multi-scale subtraction network,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Strasbourg, France: Springer, 2021, pp. 120–130, DOI: 10.1007/978-3-030-87193-2_12.
  * (43) B. Murugesan, K. Sarveswaran, S. M. Shankaranarayana, K. Ram, J. Joseph, and M. Sivaprakasam, “Psi-Net: Shape and boundary aware joint multi-task deep network for medical image segmentation,” in _Engineering in Medicine and Biology Society_. Germany, Germany: IEEE, 2019, pp. 7223–7226, DOI: 10.1109/EMBC.2019.8857339.
  * (44) R. Wang, S. Chen, C. Ji, J. Fan, and Y. Li, “Boundary-aware context neural network for medical image segmentation,” _Medical Image Analysis_ , vol. 78, p. 102395, 2022, DOI: 10.1016/j.media.2022.102395.
  * (45) Y. Fang, C. Chen, Y. Yuan, and K.-y. Tong, “Selective feature aggregation network with area-boundary constraints for polyp segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Shenzhen, China: Springer, 2019, pp. 302–310, DOI: 10.1007/978-3-030-32239-7_34.
  * (46) Y. Shen, X. Jia, and M. Q.-H. Meng, “Hrenet: A hard region enhancement network for polyp segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Strasbourg, France: Springer, 2021, pp. 559–568, DOI: 10.1007/978-3-030-87193-2_53.
  * (47) G.-P. Ji, L. Zhu, M. Zhuge, and K. Fu, “Fast camouflaged object detection via edge-based reversible re-calibration network,” _Pattern Recognition_ , vol. 123, p. 108414, 2022, DOI: https://doi.org/10.1016/j.patcog.2021.108414.
  * (48) D.-P. Fan, G.-P. Ji, T. Zhou, G. Chen, H. Fu, J. Shen, and L. Shao, “Pranet: Parallel reverse attention network for polyp segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Lima, Peru: Springer, 2020, pp. 263–273, DOI: https://doi.org/10.1007/978-3-030-59725-2_26.
  * (49) T.-C. Nguyen, T.-P. Nguyen, G.-H. Diep, A.-H. Tran-Dinh, T. V. Nguyen, and M.-T. Tran, “Ccbanet: Cascading context and balancing attention for polyp segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Strasbourg, France: Springer, 2021, pp. 633–643, DOI: 10.1007/978-3-030-87193-2_60.
  * (50) M. Cheng, Z. Kong, G. Song, Y. Tian, Y. Liang, and J. Chen, “Learnable oriented-derivative network for polyp segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Strasbourg, France: Springer, 2021, pp. 720–730, DOI: 10.1007/978-3-030-87193-2_68.
  * (51) T. Kim, H. Lee, and D. Kim, “Uacanet: Uncertainty augmented context attention for polyp segmentation,” in _Multimedia_. Chengdu, China: ACM, 2021, pp. 2167–2175, DOI: 10.1145/3474085.3475375.
  * (52) F. Shamshad, S. Khan, S. W. Zamir, M. H. Khan, M. Hayat, F. S. Khan, and H. Fu, “Transformers in medical imaging: A survey,” [Online], 2022, available: https://arxiv.org/abs/2201.09873.
  * (53) Y. Zhang, H. Liu, and Q. Hu, “Transfuse: Fusing transformers and cnns for medical image segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Strasbourg, France: Springer, 2021, pp. 14–24, DOI: 10.1007/978-3-030-87193-2_2.
  * (54) S. Li, X. Sui, X. Luo, X. Xu, Y. Liu, and R. S. M. Goh, “Medical image segmentation using squeeze-and-expansion transformers,” in _International Joint Conference on Artificial Intelligence_. Montreal, Canada: IJCAI, 2021, DOI: 10.24963/ijcai.2021/112.
  * (55) W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pvt v2: Improved baselines with pyramid vision transformer,” _Computational Visual Media_ , vol. 8, no. 3, pp. 415–424, 2022, DOI: 10.1007/s41095-022-0274-8.
  * (56) B. Dong, W. Wang, D.-P. Fan, J. Li, H. Fu, and L. Shao, “Polyp-pvt: Polyp segmentation with pyramid vision transformers,” [Online], 2021, available: https://arxiv.org/abs/2108.06932.
  * (57) D.-P. Fan, G.-P. Ji, G. Sun, M.-M. Cheng, J. Shen, and L. Shao, “Camouflaged object detection,” in _Conference on computer vision and pattern recognition_. Seattle, WA, USA: IEEE, 2020, pp. 2777–2787, DOI: 10.1109/CVPR42600.2020.00285.
  * (58) U. Ramer, “An iterative procedure for the polygonal approximation of plane curves,” _Computer graphics and image processing_ , vol. 1, no. 3, pp. 244–256, 1972, DOI: 10.1016/S0146-664X(72)80017-0.
  * (59) D.-P. Fan, J. Zhang, G. Xu, M.-M. Cheng, and L. Shao, “Salient objects in clutter,” _Transactions on pattern analysis and machine intelligence_ , pp. 1–1, 2022, DOI: 10.1109/TPAMI.2022.3166451.
  * (60) D.-P. Fan, Z. Lin, Z. Zhang, M. Zhu, and M.-M. Cheng, “Rethinking rgb-d salient object detection: Models, data sets, and large-scale benchmarks,” _Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 5, pp. 2075–2089, 2020, DOI: 10.1109/TNNLS.2020.2996406.
  * (61) X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Conference on computer vision and pattern recognition_. Salt Lake City, UT, USA: IEEE, 2018, pp. 7794–7803, DOI: 10.1109/CVPR.2018.00813.
  * (62) Y. Gu, L. Wang, Z. Wang, Y. Liu, M.-M. Cheng, and S.-P. Lu, “Pyramid constrained self-attention network for fast video salient object detection,” in _AAAI Conference on Artificial Intelligence_ , vol. 34. New York, New York, USA: AAAI Press, 2020, pp. 10 869–10 876, DOI: 10.1609/aaai.v34i07.6718.
  * (63) L. Guo, J. Liu, X. Zhu, P. Yao, S. Lu, and H. Lu, “Normalized and geometry-aware self-attention network for image captioning,” in _Conference on computer vision and pattern recognition_. Seattle, WA, USA: IEEE, 2020, pp. 10 327–10 336, DOI: 10.1109/CVPR42600.2020.01034.
  * (64) J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” in _NIPS 2016 Deep Learning Symposium_. Barcelona, Spain: Curran Associates, Inc., 2016.
  * (65) S.-H. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr, “Res2net: A new multi-scale backbone architecture,” _Transactions on pattern analysis and machine intelligence_ , vol. 43, no. 2, pp. 652–662, 2019, DOI: 10.1109/TPAMI.2019.2938758.
  * (66) S. Liu, D. Huang _et al._ , “Receptive field block net for accurate and fast object detection,” in _European conference on computer vision_. Munich, Germany: Springer, 2018, pp. 385–400, DOI: 10.1007/978-3-030-01252-6_24.
  * (67) K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Conference on computer vision and pattern recognition_. Las Vegas, NV, USA: IEEE, 2016, pp. 770–778, DOI: 10.1109/CVPR.2016.90.
  * (68) P. Krähenbühl and V. Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” in _Advances in neural information processing systems_ , vol. 24. Granada, Spain: Curran Associates, Inc., 2011, pp. 109–117.
  * (69) X. Lu, W. Wang, C. Ma, J. Shen, L. Shao, and F. Porikli, “See more, know more: Unsupervised video object segmentation with co-attention siamese networks,” in _Conference on computer vision and pattern recognition_. Long Beach, CA, USA: IEEE, 2019, pp. 3623–3632, DOI: 10.1109/CVPR.2019.00374.
  * (70) T. Zhou, J. Li, S. Wang, R. Tao, and J. Shen, “Matnet: Motion-attentive transition network for zero-shot video object segmentation,” _Transactions on image processing_ , vol. 29, pp. 8326–8338, 2020, DOI: 10.1109/TIP.2020.3013162.
  * (71) R. Liu, Z. Wu, S. Yu, and S. Lin, “The emergence of objectness: Learning zero-shot segmentation from videos,” in _Advances in neural information processing systems_. [Online]: Curran Associates, Inc., 2021.
  * (72) M. Zhang, J. Liu, Y. Wang, Y. Piao, S. Yao, W. Ji, J. Li, H. Lu, and Z. Luo, “Dynamic context-sensitive filtering network for video salient object detection,” in _International conference on computer vision_. [Online]: IEEE, 2021, pp. 1553–1563, DOI: 10.1109/ICCV48922.2021.00158.
  * (73) G.-P. Ji, K. Fu, Z. Wu, D.-P. Fan, J. Shen, and L. Shao, “Full-duplex strategy for video object segmentation,” in _International conference on computer vision_. [Online]: IEEE, 2021, pp. 4922–4933, DOI: 10.1109/ICCV48922.2021.00488.
  * (74) R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, “Frequency-tuned salient region detection,” in _Conference on computer vision and pattern recognition_. Miami, FL, USA: IEEE, 2009, pp. 1597–1604, DOI: 10.1109/CVPR.2009.5206596.
  * (75) D.-P. Fan, G.-P. Ji, X. Qin, and M.-M. Cheng, “Cognitive vision inspired object segmentation metric and loss function,” _SCIENTIA SINICA Informationis_ , vol. 6, p. 6, 2021, DOI: 10.1155/2017/4037190.
  * (76) M.-M. Cheng and D.-P. Fan, “Structure-measure: A new way to evaluate foreground maps,” _International journal of computer vision_ , vol. 129, no. 9, pp. 2622–2638, 2021, DOI: 10.1007/s11263-021-01490-8.
  * (77) R. Margolin, L. Zelnik-Manor, and A. Tal, “How to evaluate foreground maps?” in _Conference on computer vision and pattern recognition_. Columbus, OH, USA: IEEE, 2014, pp. 248–255, DOI: 10.1109/CVPR.2014.39.
  * (78) A. Borji, M.-M. Cheng, H. Jiang, and J. Li, “Salient object detection: A benchmark,” _Transactions on image processing_ , vol. 24, no. 12, pp. 5706–5722, 2015, DOI: 10.1109/TIP.2015.2487833.
  * (79) D.-P. Fan, M.-M. Cheng, Y. Liu, T. Li, and A. Borji, “Structure-measure: A new way to evaluate foreground maps,” in _International conference on computer vision_. Venice, Italy: IEEE, 2017, pp. 4548–4557, DOI: 10.1109/ICCV.2017.487.
  * (80) D.-P. Fan, C. Gong, Y. Cao, B. Ren, M.-M. Cheng, and A. Borji, “Enhanced-alignment measure for binary foreground map evaluation,” in _International Joint Conference on Artificial Intelligence_. Stockholm, Sweden: IJCAI, 2018, pp. 698–704, DOI: 10.24963/ijcai.2018/97.
  * (81) D.-P. Fan, G.-P. Ji, M.-M. Cheng, and L. Shao, “Concealed object detection,” _Transactions on pattern analysis and machine intelligence_ , pp. 1–1, 2021, DOI: 10.1109/TPAMI.2021.3085766.
  * (82) G.-P. Ji, D.-P. Fan, Y.-C. Chou, D. Dai, A. Liniger, and L. Van Gool, “Deep gradient learning for efficient camouflaged object detection,” _Machine Intelligence Research_ , 2022.
  * (83) X. Guo, J. Liu, and Y. Yuan, “Semantic-oriented labeled-to-unlabeled distribution translation for image segmentation,” _Transactions on Medical Imaging_ , vol. 41, no. 2, pp. 434–445, 2021, DOI: 10.1109/TMI.2021.3114329.
  * (84) I. B. Senkyire and Z. Liu, “Supervised and semi-supervised methods for abdominal organ segmentation: A review,” _International Journal of Automation and Computing_ , vol. 18, no. 6, pp. 887–914, 2021, DOI: 10.1007/s11633-021-1313-0.
  * (85) K. Zou, X. Yuan, X. Shen, M. Wang, and H. Fu, “Tbrats: Trusted brain tumor segmentation,” [Online], 2022, available: https://arxiv.org/abs/2206.09309.