WGMwgm QEqe EPep PMSpms BECbec DEde

[type=author, auid=000,bioid=1, orcid=0000-0002-9896-8727]

[type=author, auid=000,bioid=1, orcid=0000-0003-1313-3542]

[cor1]Corresponding authors

# Real-time Instance Segmentation of Surgical Instruments using Attention and
Multi-scale Feature Fusion

Juan Carlos Ángeles-Cerón  Gilberto Ochoa-Ruiz<EMAIL_ADDRESS>Leonardo
Chang  Sharib Ali<EMAIL_ADDRESS>Tecnologico de Monterrey, Escuela de
Ingeniería y Ciencias, México  Institute of Biomedical Engineering (IBME),
Department of Engineering Science, University of Oxford, Oxford, UK  Oxford
NIHR Biomedical Research Centre, University of Oxford, Oxford, UK

###### Abstract

Precise instrument segmentation aid surgeons to navigate the body more easily
and increase patient safety. While accurate tracking of surgical instruments
in real-time plays a crucial role in minimally invasive computer-assisted
surgeries, it is a challenging task to achieve, mainly due to 1) complex
surgical environment, and 2) model design with both optimal accuracy and
speed. Deep learning gives us the opportunity to learn complex environment
from large surgery scene environments and placements of these instruments in
real world scenarios. The Robust Medical Instrument Segmentation 2019
challenge (ROBUST-MIS) provides more than 10,000 frames with surgical tools in
different clinical settings. In this paper, we use a light-weight single stage
instance segmentation model complemented with a convolutional block attention
module for achieving both faster and accurate inference. We further improve
accuracy through data augmentation and optimal anchor localisation strategies.
To our knowledge, this is the first work that explicitly focuses on both real-
time performance and improved accuracy. Our approach out-performed top team
performances in the ROBUST-MIS challenge with over 44% improvement on area-
based multi-instance dice metric MI_DSC and 39% on distance-based multi-
instance normalized surface dice MI_NSD. We also demonstrate real-time
performance ($>60$ frames-per-second) with different but competitive variants
of our final approach.

###### keywords:

Deep learning
MIS instance segmentation
Real-time instance segmentation
Single-stage segmentation
Attention Modules
Multi-scale feature fusion

##  1 Introduction

Surgical site infection (SSI) has been the most common cause of hospital-
acquired infection and the most common way of infection transmission in
patients undergoing surgery Gandaglia et al. (2014); Caroff et al. (2019). It
is therefore imminently important to develop strategies for reducing such
infection rates. Minimally invasive surgical (MIS) procedures compared to open
surgery lowers such risks Caroff et al. (2019). For these reasons and due to
the growth the data science in the operating room applications, there exists
an increasing demand of computer assisted/robotic surgery to improve the
efficacy of MIS Bartoli et al. (2012); Sheetz et al. (2020).

Computer-assisted minimally invasive surgery such as endoscopy has grown in
popularity in recent years. However, due to the nature of these procedures,
issues like limited field-of-view, extreme lighting conditions, lack of depth
information and difficulty in manipulating operating instruments demand
strenuous amounts of effort from the surgeons Roß et al. (2021). Surgical data
science applications Maier-Hein et al. (2021a) could provide physicians with
context-aware assistance during minimally invasive surgery in order to
overcome these limitations and increase patient safety.

One of the main forms of assistance is providing accurate tracking of medical
instruments using computer vision techniques, such as object detection and
localisation or instance segmentation methods Ward et al. (2021). These
systems are expected to be a crucial component in tasks ranging from surgical
navigation, skill analysis and complication prediction during surgeries, as
well as other computer integrated surgery (CIS) applications Maier-Hein et al.
(2021b); Fu et al. (2021). Nonetheless, methods for accurate tracking of
instruments are often deployed in difficult operational scenarios in which the
presence of bleeding, over or under exposed frames, smoke, reflection and
other types of artifacts are oftentimes unavoidable Bodenstedt et al. (2018).
The net effect of these issues increases the missed detection rates in
endoscopic surveillance and in general, it limits the overall robustness of
computer vision algorithms, hampering the adoption of AI-based tools in this
context Ali et al. (2021). Moreover, real-time deployment of such tools is of
tremendous value and one of the major requirement for it to be applied in
clinical setting. Therefore, the development of robust and real-time
techniques that can be effectively deployed during real endoscopic
interventions is of utmost importance. In this regard, recent years have seen
a significant increase in the number of computer vision contests geared
towards endoscopy. More specifically, the Robust Medical Instrument
Segmentation (ROBUST-MIS) Challenge Roß et al. (2021) at the International
Conference on Medical Image Computing and Computer Assisted Interventions
(MICCAI) has sought to address some of the issues discussed above. This
challenge represents an important and necessary effort to encourage the
development of robust models for surgical instrument segmentation, integrating
the developments in computer-assisted surgeries, and benchmark the
generalization capabilities of the developed methods on different clinical
scenarios.

Furthermore, the challenge organizers provide a large high-quality dataset in
an effort to overcome one of the major bottlenecks of the development of
robust methodologies, i.e., lack of annotated multi-instance instrument
segmentation data. The development of surgical tool navigation and tracking
methods in a complex environment will enable improved patient care during
surgery by maximising the focus of surgeons. It will also accelerate research
in the direction of robotic surgery.

Previous approaches for instance segmentation submitted to the ROBUST-MIS
challenge have been mostly based on two-stage detectors such as Mask R-CNN He
et al. (2018). While these models exhibited decent performances in terms of
robustness, they suffer from high inference times due to well-known
architectural limitations of such models, preventing them for achieving real-
time performance (i.e. mean average test time of only around 5 frames-per-
second, fps). However, real-time performance is mandatory in order to fully
exploit the capabilities of tracking applications in surgeries. While deep
learning methods using lightweight models are also available, they fail to
robustly segment objects in endoscopy imaging Ali et al. (2020). Similarly,
there is a trend in using ensemble models to improve the overall segmentation
accuracy in these images. However, combining few models together drastically
increases the inference time, thereby making it less feasible to deploy such
models in clinical settings Xu et al. (2020).

In order to overcome the current inference limitations, while maintaining a
robust performance in terms of tool segmentation results, we propose a new
approach based on the single-stage model for instance segmentation. Although
recent years have seen an steady increase in the search for more capable one
stage detectors and instance segmentation architectures (for example PolyYolo
Hurtik et al. (2020), BlendMask Chen et al. (2020) and Solov2 Wang et al.
(2020)), the majority of these models have not been used in the context of
endoscopic computer vision due to the performance gap they still present. The
YOLACT++ Bolya et al. (2020) architecture is one of the most recent methods
for real-time instance segmentation and it is particularly appealing due to
its simplified architecture capable of learning to localize instance masks
automatically with minimum computational overhead. It does so by generating a
dictionary of non-local prototype masks over the entire image and predicting a
set of linear combination coefficients per instance. Thus, for this
contribution we have used YOLACT++ as a baseline architecture upon which we
have developed several improvements to make it suitable for robust surgical
instrument instance segmentation. To this extend, we have explored the use of
attention modules on the outputs of the network’s backbone and feature pyramid
network (FPN) at multiple scales. Moreover, we have additionally carried out a
series of optimization techniques by analysing the worst-performing frames of
the best model in our experiments. The optimization techniques includes
stronger data augmentation, anchor optimization, and multi-scale feature
fusion.

Our main contributions include:

  * •

A real-time single-stage instance segmentation framework with attention
mechanisms

  * •

Use of domain-targeted data augmentation

  * •

Anchor box optimization via a differential evolution search algorithm Zlocha
et al. (2019)

  * •

Exploiting global contextual features by integrating multi-scale fusion in the
network’s backbone

  * •

Thorough analysis of the worst-case samples to evaluate each of the tested
model

The rest of the paper is organized as follows. In Section 2 we present
previously published work related to medical instrument segmentation, instance
segmentation methods, attention mechanisms and multi-scale feature fusion
network. In Section 3, we present the details of the ROBUST-MIS dataset and
our proposed approach for surgical instrument segmentation. Section 4 presents
our data preparation, experimental setup and results. Next, in Section 5 we
discuss the effects of the different types of network configurations and
future directions. Finally, Section 6 concludes the paper.

##  2 Related work

In this section, we will discuss some of the most important aspects to
understand the proposed contribution, namely: instance segmentation and its
current limitations, recent works in attention mechanisms, anchor box
optimization techniques specifically tailored to the addressed problem and
finally, multi-scale fusion networks that followed to make our extended
instance segmentation model more robust.

###  2.1 Deep learning for instrument segmentation

Deep learning has accelerated research for surgical instrument segmentation
and the public access of labelled data via instrument segmentation challenges
mostly at EndoVis (refer to Allan et al. (2019), Allan et al. (2020), Roß et
al. (2021)) have contributed to these developments over recent years. Built
upon the UNet Ronneberger et al. (2015a), LinkNet and TernausNet was developed
for instrument segmentation on robotic surgery dataset Shvets et al. (2018),
acquired by da Vinci Xi surgical system of several different porcine
procedures made available in EndoVis17 Allan et al. (2019). Milletari et al.
(2018) proposed a convolutional long short term memory (LSTM) with deep
residual networks in coarse-to-fine strategy showing greater improvements over
other state-of-the-art approaches including UNet Ronneberger et al. (2015a)
and FCN Long et al. (2015a) on EndoVis 2015 instrument segmentation challenge
dataset111https://endovissub-instrument.grand-challenge.org/EndoVisSub-
Instrument/ focused on laparoscopic and robotic surgery.

The recent “Robotic Instrument Segmentation Sub-Challenge” introduced also at
EndoVis was oriented towards binary segmentation, sub-component segmentation
and instrument identification and instance segmentation tasks Roß et al.
(2021). The challenge was focused to assess the robustness and generalization
capabilities of the deep learning models. Most of the competing methods in
this challenge have been mostly based on Mask-RCNN He et al. (2017)
implementations and its variants. Participants also explored methods such as
OR-UNet Isensee and Maier-Hein (2020), DeepLabV3+ Chen et al. (2018), U-Net
Ronneberger et al. (2015b) and RASNet Ni et al. (2019). The best performing
methods for the binary segmentation task were OR-UNet and DeepLabv3+ with pre-
trained ImageNet encoders. Some other contestants also explored the use of
ensemble methods, but they were typically limited in speed, and thus some of
the most robust methods are incapable of attaining the real-time performances
required for realistically segmenting and tracking objects on endoscopic video
data. A double decoder-encoder network was explored for faster binary mask
segmentation on ROBUST-MIS’19 daataset Jha et al. (2021) that outperformed
several state-of-the-art methods. Recently, a one-shot instrument segmentation
method Zhao et al. (2021) using anchor guided meta-learning approach was
proposed and validated on several publicly available datasets.

###  2.2 Real-time instance segmentation methods

While extensive research has been conducted for the development of real-time
object detection and semantic segmentation models, few works have tackled the
problem of real-time instance segmentation He et al. (2017); Bolya et al.
(2019). This is due to the increased complexity in the instance segmentation
task that requires prediction of instance labels and pixel-level segmentation
simultaneously. One-stage methods Hurtik et al. (2020); Lee and Park (2019);
Chen et al. (2020) though conceptually faster than two-stage methods (e.g.,
Mask RCNN He et al. (2017)), still require many non-trivial computations
(e.g., mask voting). This severely limits their speed making them not
unsuitable for real-time applications. In contrast, recent methods Bolya et
al. (2019, 2020) make use of lightweight assembly of masks (only a linear
combination is used), making the approach very efficient. Although YOLACT
Bolya et al. (2019) was one of the first real-time one-stage instance
segmentation approach, the accuracy gap compared to Mask R-CNN He et al.
(2017) was still significant. While Mask R-CNN is based on a two-stage object
detector (e.g., Faster R-CNN Ren et al. (2015)), YOLACT Bolya et al. (2019) is
built on one-stage detector (RetinaNet Lin et al. (2017b)) that directly
predicts boxes without proposal step, limiting its accuracy. This was
partially addressed with the introduction of Yolact++ Bolya et al. (2020)
incorporates deformable convolutions into the backbone network, improving the
feature sampling and yielding an improved accuracy. Furthermore, the
prediction head was optimized with better anchor scale and aspect ratio
choices for an increased object recall.

###  2.3 Attention mechanisms

Attention has been able to boost model performance across a wide range of
computer vision tasks, such as image captioning You et al. (2016), visual
question answering Xu et al. (2016), and visual attribute prediction Seo et
al. (2018). Attention allows the network to focus on the most relevant
features without the need of additional supervision, preventing redundant use
of information and extracting salient features that are useful for a given
task. Attention mechanisms enable convolutional neural networks to overcome
the size limitations of its receptive field, as it has proven to be excellent
at extracting global dependencies between inputs and outputs, thus, improving
the modeling of long-range dependencies even at opposite ends of and image
Sinha and Dolz (2021). For example, similar textures may appear in different
parts of an image, several disjoint semantic cues may provide insight to the
general classification of an image, and an object might present complex and
occluded parts throughout an image Chaudhari et al. (2019). In the context of
medical instance segmentation, multiple attention-based models Kaul et al.
(2019); Gu et al. (2020); Sinha and Dolz (2021) have obtained state-of-the-art
performance in fields like brain tumor, skin cancer and lung lesion
segmentation on CT scans and X-rays. Until now, instance segmentation of
medical instruments in laparoscopic surgeries using attention mechanisms has
not been fully explored.

###  2.4 Multi-scale feature fusion

Due to the wide range of scale variation of objects found in instance
segmentation, multi-scale features are essential for robust performance Wang
et al. (2020). Therefore, multi-scale feature aggregation is an adequate
strategy to create detailed parsing maps Ding et al. (2018). Current methods
address this issue by using encoder-decoder architectures Ronneberger et al.
(2015a); Lin et al. (2017a) that combine high level and low level features at
a single scale Long et al. (2015b) or multiple scales Ronneberger et al.
(2015a). However, these approaches suffer from redundant use of information
Sinha and Dolz (2021). In the field of medical instrument segmentation, the
use of multi-scale aggregation has not been fully explored, especially
combined with attention for improved robustness.

##  3 Materials and Method

In this section we present details on the dataset used in our study and
describe our proposed framework for multi-instance surgical instrument
segmentation.

###  3.1 The ROBUST-MIS challenge dataset

Table 1: Training and test sample distribution for each both training and test stages of the ROUBUST-MIS challenge Roß et al. (2021). The quantities in parenthesis represent the % of frames with no instrument instance. 2*Procedure |  2*Training | Testing
---|---|---
|  | Stage 1 | Stage 2 | Stage 3
| Procto-
---
colectomy
2,943 (2%) | 325 (11%) | 255 (11%) | 0
| Rectal
---
resection
3,040 (20%) | 338 (20%) | 289 (15%) | 0
| Sigmoid
---
resection*
0 | 0 | 0 |  | 2,880
---
(23%)
TOTAL | 5,983 (17%) | 663 (15%) | 514 (13%) |  | 2,880 (23%)
---
*Unknown surgery

For our experiments we made use of the Robust Medical Instrument Segmentation
(ROBUST-MIS) Maier-Hein et al. (2021b) challenge dataset which is the first
large-scale annotated MIS dataset. The dataset is comprised of a total of
10,040 annotated video frames extracted from 30 minimally invasive daily-
routine surgical procedures and includes detailed segmentation ground truth
masks for the surgical instruments present in these frames.

The surgical procedures include 10 rectal resection procedures, 10
proctocolectomy procedures, and 10 sigmoid resection procedures. The image
resolution of all the provided frames is $960\times 540$ pixels. In order to
measure the robustness test, the dataset is comprised of three unique test
sets and divided into different stages:

  1. Stage 1: Test data taken from the same procedures from which the training data were extracted

  2. Stage 2: Test data taken from the exact same type of surgery as the training data but from procedures (patients) not included in the training

  3. Stage 3: Test data taken from a different (unseen) but similar type of surgery and different (unseen) patients

The detailed training and test distribution is summarised in Table 1. However,
for testing we have used only test set 3, i.e. Stage 3, which is from an
unseen Sigmoid resection procedure and allows us to validate on the
generalisability of the proposed framework directly. Sample images for
challenging frames provided in the test set 3 is shown in Figure 1.

Figure 1: Challenging images present in test dataset. Stage 3 test data
consisted of several frames that consisted of a) instrument flare, b) parital
occlusions due to blood, c) occlusion due to smoke, d) underexposed regions
with instrument, e) motion blur, f) multiple different instruemnts in the
scene, g) partial occlusion due to organ, h) transparent instrument and i)
different instruments crossing.

###  3.2 Method

Figure 2: Overview of our proposed framework. Our architecture is built upon
popular single-stage YOLACT++ and comprises of additional multi-scale feature
fusion and attention modules. It is to be noted that the attention modules is
easily interchangeable from criss-cross attention (CCAM) to convolutional
block attention module (CBAM) or none.

This section describes our proposed approach for multi-instance segmentation
of medical instruments in the ROBUST-MIS dataset. Our proposed framework (see
Figure 2) is built upon the single-stage YOLACT++ Bolya et al. (2020) instance
segmentation architecture. For our framework we employ ResNet-101 He et al.
(2015) as the backbone network, followed by a multi-scale feature fusion
(MSFF) module used to aggregate contextual information from the feature maps
across all scales (i.e. high-to-low resolution feature representations). Each
of these contextually rich fused features are then passed through attention
modules to further refine these representations before being forwarded to the
feature pyramid network (FPN) Lin et al. (2017c). A second set of attention
modules are then applied to further enhance the FPN output features allowing
for an improved performance of the prototype network and our anchor-optimized
prediction head. Finally, we perform classical non-maximum suppression for
final mask instance prediction which is then combined with the prototype mask
and the cropping provides the predicted bounding box.

Below we describe our MSFF module, attention mechanisms, and the anchor
optimization used our framework.

####  3.2.1 Multi-scale feature fusion

To aggregate multi-scale features, while maintaining a high-resolution
representation, we integrated a fusion module inspired by the method proposed
in Wang et al. (2018). Considering the features at different scales indicated
as $F_{s}$ where $s$ denote the scale level in the architecture (see Figure
3), features from each level $s$ are up-sampled through transposed convolution
to the size of the highest resolution feature maps in the architecture,
leading to enlarged feature maps $F^{\prime}_{s}$. Next, all $F^{\prime}_{s}$
are concatenated into a single tensor which is passed through a convolutional
layer to integrate context from all scales into a single feature map
$F_{MS}=conv([F^{\prime}_{0},F^{\prime}_{1},F^{\prime}_{2},F^{\prime}_{3},F^{\prime}_{4}])$.
In this manner, $F_{MS}$ encodes both low-level and high-level semantics
learned at different stages. Finally, $F_{MS}$ is concatenated with each of
the $F^{\prime}_{s}$ feature maps and convolved to aggregate multi-scale
information, creating multi-scale fused feature maps $F_{A}$.

Figure 3: Multi-scale feature fusion (MSFF) module. Feature maps from the
different scale feature maps of the backbone are combined to create aggregated
features from all scales.

Note that multi-scale feature fusion can be applied in the same way in both
the backbone features and the FPN features. We opted to attach it on the
backbone features as we believe it would create stronger representation which
will get even further refined on the FPN.

####  3.2.2 Attention mechanisms

We employed Criss-cross Attention Modules (CCAM) Huang et al. (2020) and
Convolutional Block Attention Modules (CBAM) Woo et al. (2018), specifically
because of their fast and computationally efficient performance, which is
paramount to introduce as less computational overhead as possible into the
model, and therefore maintain low inference times. We attach the attention
modules between the backbone and neck structures, as well as the neck and head
of our network (see Figure 2). The rationale behind the selection of these
locations is that the addition of attention allows the model to extract richer
context by aggregating local information with its corresponding global
dependencies Huang et al. (2020). Additionally, attention aids to emphasize
interdependent relationships between channel maps and between spatial regions
without additional supervision Woo et al. (2018). Since the backbone network
and the FPN are where most of the semantic context is distilled, it is a
natural choice to attempt to refine their feature representations using
attention, especially since mask prototypes and the prediction head benefit
from better features.

####  3.2.3 Loss function and anchor optimisation

We use a combination of four losses: classification loss $L_{cls}$, bounding
box regression loss $L_{box}$, mask loss $L_{mask}$, and semantic segmentation
loss $L_{seg}$ with weights of 1, 1.5, 6.125 and 1, respectively. We employ
softmax cross entropy for $L_{cls}$ with 1 positive label (instrument) and 1
background label. In the case of $L_{box}$ we used smooth-$L_{1}$ loss in the
same way as Liu et al. (2016). $L_{mask}$ is defined as the pixel-wise binary
cross entropy between the predicted masks and the ground truth. $L_{seg}$ is
applied on layers that are only evaluated during training for additional
feature richness Bolya et al. (2020).

We further optimized the anchor boxes Zlocha et al. (2019) by defining 5
target scales and aspect ratios considering the size of our rescaled images
($600\times 600$ pixels). The resulting scales were [0.435, 0.502, 0.578,
0.664, 0.762] and the resulting aspect ratios [0.267, 0.554, 1.0, 1.804,
3.746].

##  4 Experiments and Results

###  4.1 Data preparation

As shown in Table 1, the ROBUST-MIS dataset contains about 17% empty frames
(ef) on its training set. These frames do not have any visible instruments in
them, and although we could have left them as negative examples for training,
we opted to remove them from the training set. This decision was taken
considering that the data already has plenty of negative examples in the
frames’ background for the model to learn from; additionally, removing such
frames speeds up the training. In the end, a total of 996 images with no
visible instruments were discarded, leaving 4,987 frames in our training set.
We then applied an 85-15% split to the curated training set with 15% for
validation purposes. The training set was then randomly shuffled before
creating the train and validation splits. We finally obtained a training set
composed of 4,239 frames and a validation set comprised of 748 frames. As a
final step, the training and validation datasets were converted to COCO-style
for our instance segmentation framework, which involved extracting mask
contours and generating bounding box coordinates from the provided annotation
images and translating them to the target JSON format.

###  4.2 Training setup

The training was performed on an NVIDIA DGX-1 system consisting of 8 NVIDIA
Volta-based GPUs; however, each model was trained on _a single_ GPU. The
models were trained for up to 400,000 iterations with a learning rate of
0.001, momentum of 0.9, weight decay of $5e^{-4}$, and a batch size of 16.
Input images were resized to $600\times 600$ pixels during training, as this
configuration constituted the best trade-off between GPU memory usage and
segmentation performance. We applied data augmentation techniques to increase
the model’s performance. These included random photometric distortions (i.e.,
changes in contrast, color-space, saturation, hue, brightness, and noise
transformations) and affine transformations (i.e., random scaling, random
sample crop, and random mirror).

###  4.3 Ablation study setup

| Model
---
Identifier
| Attention
---
Type
| Backbone
---
Attention
| FPN
---
Attention
Base YOLACT++ |  |  |
CCAM-Backbone | CCAM | ✓ |
CCAM-FPN | CCAM |  | ✓
CCAM-Full | CCAM | ✓ | ✓
CBAM-Backbone | CBAM | ✓ | ✓
CBAM-FPN | CBAM |  | ✓
CBAM-Full | CBAM | ✓ | ✓
| CBAM-Full
---
\+ Aug
CBAM | ✓ | ✓
| CBAM-Full
---
\+ Aug + Anch
CBAM | ✓ | ✓
| CBAM-Full
---
\+ Aug + Anch + MSFF
CBAM | ✓ | ✓
|  |  |

Table 2: Model configurations for our experiments with integration of
different attention mechanisms in unique settings.

Our experiments systematically integrate attention mechanisms in two strategic
locations of the baseline YOLACT framework: 1) at the output of each
convolutional block of the ResNet-101 backbone He et al. (2015), and 2) at the
multi-scale output features of the FPN Lin et al. (2017c). The incorporation
of attention in these locations was alternated throughout experiments leading
to three different network configurations:

  1. 1.

Exclusively incorporated in the backbone

  2. 2.

Exclusively incorporated in the FPN

  3. 3.

Integrated in the backbone and FPN simultaneously (which we refer to as a
_Full_ configuration)

At the end, six attention-based models were created by following this
strategy, plus a baseline network without attention. Next, we selected the top
performing configuration from the six mentioned network configurations. We
then applied other optimization techniques to understand the network
performance that included stronger data augmentation and anchor optimization.
Finally, we added our multi-scale MSFF module. Table 4.3 summarizes all the
different model configurations.

###  4.4 Metrics and assessment

The algorithms’ performance was evaluated following the guidelines defined by
the ROBUST-MIS Challenge. Robustness performance was assessed considering the
area-based metric multi-instance dice MI_DSC and the distance-based multi-
instance normalized surface dice MI_NSD using the code implementations
employed in the challenge which are provided in (Roß and Reinke, ).
Furthermore, our reported model rankings were computed using the publicly
available  _challengeR_ Wiesenfarth et al. (2021) R package developed by the
challenge organizers to accurately evaluate competitors. The ranking stability
was investigated using bootstrapping for quantifying ranking variability using
1000 samples.

The robustness rankings are particularly focused on models’ capabilities in
stage 3 of the challenge and pay particular attention on the worst-case
performance of methods. For this reason, the robustness rankings are computed
by aggregating the resulting scores for all the test cases by the 5%
percentile instead of by the mean or median.

We performed inference speed assessments by running inference tests on a 10
second video snippet from the ROBUST-MIS dataset a total of ten times per
model. The reported frame rates were then aggregated by the mean. Inference
was tested on a _single_ Tesla P100 GPU from the DGX-1 cluster with video
multi-frame enabled.

Figure LABEL:fig:1a shows the qualitative comparison between worst frames of
_Base YOLACT++_ and their corresponding frames from _CBAM-Full_. We can
observe from the images on the left side that the baseline model often
presents missed detections, which hinders the model’s recall. Similarly, the
model confuses tissue and other objects like bandages as instruments,
evidencing its lack of robustness. On the other hand, _CBAM-Full_ overcomes
some of these problems by attending to the important features, leading to
better localization and segmentation results. We identified four different
conditions that seemed particularly challenging for _CBAM-Full_ : transparent
instruments, vertical instruments, small instruments on the edge of the field
of view, and partially occluded instruments. According to Roß et al. (2021),
most of these issues have also been challenging to previous participants.
Figure LABEL:fig:1b illustrates the comparison of worst frames of _CBAM-Full_
and the improvements obtained after training with stronger data augmentation.

In contrast to the baseline model with attention, we can observe that _CBAM-
Full + Aug_ is capable of addressing problematic instances such as small
instruments on the edge of the field of view. Similarly, transparent,
partially occluded, and vertical instruments are now detected and segmented to
a larger extent.

Howver, _CBAM-Full + Aug_ still presented recurrent missed detections on long
vertical and transparent instruments. Figure LABEL:fig:1c illustrates the
comparison of worst frames of _CBAM-Full + Aug_ and the improvements obtained
after training with an optimized set of anchors. We can observe that the
anchor optimization in _CBAM-Full + Aug + Anch_ led to additional detection
and segmentation improvements not only on previously undetected objects but
across all instruments. Figure LABEL:fig:1d illustrates the comparison of
worst frames of _CBAM-Full + Aug + Anch_ and the improvements obtained after
training with MSFF.

We can observe that _CBAM-Full + Aug + Anch + MS_ is better at recognizing
more challenging instances, like small under-exposed instruments, transparent
instruments, and reflections.

Figure 6: Network output with and without attention. (top) Large part of image
covered with instrument in purple and in blue (left). (bottom) Three
instruments two at the bottom of image and one on the top of image (see left).

##  5 Discussion

While deep learning has allowed us to design data driven solutions,
generalisation remains a key issue that can cause significant performance
degradation, especially in instrument segmentation applications where the
intervention objects are of variable shapes and exposed to dynamic
environments (e.g., smoke, flares, specularity etc, see Figure 1). In surgery,
real-time performance of tool segmentation is of enormous importance for
clinical utility. However, most methods build in the past relied on two stage
networks that are not computationally efficient.

To this end, we built over single stage YOLACT Bolya et al. (2019) and added
several modifications to improve both the accuracy and robustness of our final
surgical instrument segmentation algorithm. Among our models, we observed that
those based on CBAM achieved slightly better performance that the ones based
on CCAM. Regardless, attention-integrated models always outperformed the
attention-less baseline in terms of robustness. Figure
LABEL:fig:boxplots_stage1 shows dot-and-boxplots of the metric values obtained
by each algorithm over all test cases in stage 3 dataset (unseen test dataset
in the MIS challenge).

We can observe that adding attention mechanisms boosts the performance
compared to the baseline model used in our architecture design Bolya et al.
(2019). This especially true for instances below the third quartile, which are
the most important for our performance metrics. Among the three model
variations to which we added CCAM attention modules, CCAM-Backbone achieved
the best results in terms of robustness (0.313 MI_DSC and 0.338 MI_NSD, see
Table LABEL:tab:metric_scores). This indicates that the contextually enriched
feature maps from the ResNet-101 backbone are powerful enough to generate more
accurate mask prototypes and coefficients in the YOLACT architecture, and
ultimately better segmentation outputs. However, the CBAM-Full model
outperformed all other attention modules by nearly 9% on MI_DSC over the best
performing CCAM-Backbone.

The superiority of CBAM when integrated together with backbone and FPN all
together enhances both channel-wise and block attentions better to represent
local and global features well. The FPN layer allows to capture size
variability present in the dataset for which CCAM yielded zero on aggregated
MI_DSC score (Figure LABEL:fig:boxplots_stage1).

We thoroughly investigated on model improvement for generalisability through
different mechanisms such as data augmentation and optimisation of anchor
weights. These steps were experimentally proven to be right directions giving
subsequent increase in both aggregated MI_DSC and MI_NSD (see Table
LABEL:tab:metric_scores, Figure LABEL:fig:1a and Figure LABEL:fig:1b). For
example, by using augmentation, we observed that the images with different
view points (mostly long sized instruments either straight or slightly
oblique) performed better; this can be because of low number of such samples
in the dataset (Figure LABEL:fig:1b). Similarly, for the variable size
instruments optimising anchors provided substantial improvements (Figure
LABEL:fig:1c). We also noticed issues with the small instrument appearing
either on the bottom or sides of the frame, and also for those which appeared
as background, due to specularity or covering large tissue area with similar
color (see Figure LABEL:fig:1c, left). The use of the Multi-scale feature
fusion (MSFF) network allowed us to fuse features at different scales and
layers (Figure LABEL:fig:1d).

Our architecture with MSFF network integrated together with attention maps
allowed to transfer both local and global context fusing high and low-level
feature representations. As a result, we observed further improvement of
nearly 2% on aggregated MI_DSC above the CBAM-Full with augmentation and
anchor approach (Table LABEL:tab:metric_scores). Our final best approach has
near real-time performance of 24 FPS on NVIDIA Volta GPU. However, our second
best performing method can run at 69 FPS which is above the required real-time
performance in most cases (i.e., 45 FPS). A similar, performance improvement
was observed for other two test datasets (Table LABEL:tab:suppTab1).

To better understand the behavior of the two types of attention modules
implemented in our experiments, we visualize the attention maps generated by 3
different models, in addition to the feature maps of the baseline model (see
Figure 6). We chose the _CBAM-Full_ , _CCAM-Full_ , and _CBAM-Full + Aug +
Anch + MS_ configurations for visualization purposes as they show the refined
feature maps using the two different attentions modules. Figure 6 illustrate
the activation maps taken before and after the attention modules. We visualize
the effects of attention on this specific point of the network since mask
prototypes and coefficients are extracted from FPN features and backbone
network (Full). We can observe that over-mixing problem of _CCAM-Full_. In
fact, activation maps before CCAM in the Full model are cleaner and more
discriminative than the actual attention output. This confirms that CCAM
excels at refining features that have not been mixed before. On the other
hand, CBAM-based models produce slightly lower quality feature maps from
attention modules in the backbone, as it can be observed by the clouds and
blobs on the top maps of the figure. Nonetheless, the features are drastically
refined after passing the attention modules in the Full, leading to clean and
discriminative activations. This corroborates our hypothesis that CBAM is
better at refining previously aggregated data, yielding a superior overall
performance.

Furthermore, our current work is built upon publicly available retrospective
dataset providing a strong evidence of robustness ability compared to
presented methods at the MIS challenge and current literature. In future work,
we will validate on prospective data and benchmark it in clinical settings.

##  6 Conclusion

We have developed a real-time novel architecture that builds upon a single-
stage instance segmentation method. Through our step-wise solution to
different problems in surgical data, we have identified and integrated
components that can be deal with existing challenges for robust segmentation
of surgical instruments. We have provided comprehensive experiments and
analysis that supports our final architecture development and its impact on
surgical tool segmentation in clinic. Our method outperformed all methods
reported in recently conducted ROBUST-MIS challenge.

## Acknowledgements

The authors would like to thank the AI Hub and the CIIOT at Tecnologico de
Monterrey for their support for carrying the experiments on their NVIDIA’s DGX
computer.

## References

  * Ali et al. (2021) Ali, S., Dmitrieva, M., Ghatwary, N., Bano, S., Polat, G., Temizel, A., Krenzer, A., Hekalo, A., Guo, Y.B., Matuszewski, B., Gridach, M., Voiculescu, I., Yoganand, V., Chavan, A., Raj, A., Nguyen, N.T., Tran, D.Q., Huynh, L.D., Boutry, N., Rezvy, S., Chen, H., Choi, Y.H., Subramanian, A., Balasubramanian, V., Gao, X.W., Hu, H., Liao, Y., Stoyanov, D., Daul, C., Realdon, S., Cannizzaro, R., Lamarque, D., Tran-Nguyen, T., Bailey, A., Braden, B., East, J.E., Rittscher, J., 2021\.  Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy.  Medical Image Analysis 70, 102002\.  doi:https://doi.org/10.1016/j.media.2021.102002.
  * Ali et al. (2020) Ali, S., Zhou, F., Braden, B., Bailey, A., Yang, S., Cheng, G., Zhang, P., Li, X., Kayser, M., Soberanis-Mukul, R.D., Albarqouni, S., Wang, X., Wang, C., Watanabe, S., Oksuz, I., Ning, Q., Yang, S., Khan, M.A., Gao, X.W., Realdon, S., Loshchenov, M., Schnabel, J.A., East, J.E., Wagnieres, G., Loschenov, V.B., Grisan, E., Daul, C., Blondel, W., Rittscher, J., 2020\.  An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy.  Scientific Reports 10, 2748\.  doi:10.1038/s41598-020-59413-5.
  * Allan et al. (2020) Allan, M., Kondo, S., Bodenstedt, S., Leger, S., Kadkhodamohammadi, R., Luengo, I., Fuentes-Hurtado, F., Flouty, E., Mohammed, A.K., Pedersen, M., Kori, A., Varghese, A., Krishnamurthi, G., Rauber, D., Mendel, R., Palm, C., Bano, S., Saibro, G., Shih, C.S., Chiang, H.A., Zhuang, J., Yang, J., Iglovikov, V.I., Dobrenkii, A., Reddiboina, M., Reddy, A., Liu, X., Gao, C., Unberath, M., Azizian, M., Stoyanov, D., Maier-Hein, L., Speidel, S., 2020.  2018 robotic scene segmentation challenge.  ArXiv abs/2001.11190.
  * Allan et al. (2019) Allan, M., Shvets, A.A., Kurmann, T., Zhang, Z.V., Duggal, R., Su, Y.H., Rieke, N., Laina, I., Kalavakonda, N., Bodenstedt, S., García-Peraza, L.C., Li, W., Iglovikov, V.I., Luo, H., Yang, J., Stoyanov, D., Maier-Hein, L., Speidel, S., Azizian, M., 2019.  2017 robotic instrument segmentation challenge.  ArXiv abs/1902.06426.
  * Bartoli et al. (2012) Bartoli, A., Collins, T., Bourdel, N., Canis, M., 2012\.  Computer assisted minimally invasive surgery: is medical computer vision the answer to improving laparosurgery?  Med Hypotheses 79, 858–863.
  * Bodenstedt et al. (2018) Bodenstedt, S., Allan, M., Agustinos, A., Du, X., Garcia-Peraza-Herrera, L., Kenngott, H., Kurmann, T., Müller-Stich, B., Ourselin, S., Pakhomov, D., Sznitman, R., Teichmann, M., Thoma, M., Vercauteren, T., Voros, S., Wagner, M., Wochner, P., Maier-Hein, L., Stoyanov, D., Speidel, S., 2018\.  Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery.  arXiv:1805.02475.
  * Bolya et al. (2019) Bolya, D., Zhou, C., Xiao, F., Lee, Y.J., 2019.  YOLACT: real-time instance segmentation.  CoRR abs/1904.02689.
  * Bolya et al. (2020) Bolya, D., Zhou, C., Xiao, F., Lee, Y.J., 2020.  YOLACT++: better real-time instance segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence , 1–1doi:10.1109/tpami.2020.3014297.
  * Caroff et al. (2019) Caroff, D.A., Chan, C., Kleinman, K., Calderwood, M.S., Wolf, R., Wick, E.C., Platt, R., Huang, S., 2019\.  Association of Open Approach vs Laparoscopic Approach With Risk of Surgical Site Infection After Colon Surgery.  JAMA Network Open 2, e1913570--e1913570.
  * Chaudhari et al. (2019) Chaudhari, S., Polatkan, G., Ramanath, R., Mithal, V., 2019\.  An attentive survey of attention models.  ArXiv abs/1904.02874.
  * Chen et al. (2020) Chen, H., Sun, K., Tian, Z., Shen, C., Huang, Y., Yan, Y., 2020.  Blendmask: Top-down meets bottom-up for instance segmentation.  arXiv:2001.00309.
  * Chen et al. (2018) Chen, L., Zhu, Y., Papandreou, G., Schroff, F., Adam, H., 2018.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  CoRR abs/1802.02611.
  * Ding et al. (2018) Ding, H., Jiang, X., Shuai, B., Liu, A.Q., Wang, G., 2018.  Context contrasted feature and gated multi-scale aggregation for scene segmentation, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2393--2402.  doi:10.1109/CVPR.2018.00254.
  * Fu et al. (2021) Fu, Z., Jin, Z., Zhang, C., He, Z., Zha, Z., Hu, C., Gan, T., Yan, Q., Wang, P., Ye, X., 2021\.  The future of endoscopic navigation: A review of advanced endoscopic vision technology.  IEEE Access 9, 41144--41167.  doi:10.1109/ACCESS.2021.3065104.
  * Gandaglia et al. (2014) Gandaglia, G., Ghani, K.R., Sood, A., Meyers, J.R., Sammon, J.D., Schmid, M., Varda, B., Briganti, A., Montorsi, F., Sun, M., Menon, M., Kibel, A.S., Trinh, Q.D., 2014.  Effect of Minimally Invasive Surgery on the Risk for Surgical Site Infections: Results From the National Surgical Quality Improvement Program (NSQIP) Database.  JAMA Surgery 149, 1039--1044.
  * Gu et al. (2020) Gu, R., Wang, G., Song, T., Huang, R., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T., Zhang, S., 2020\.  Ca-net: Comprehensive attention convolutional neural networks for explainable medical image segmentation.  IEEE Transactions on Medical Imaging , 1–1doi:10.1109/tmi.2020.3035253.
  * He et al. (2017) He, K., Gkioxari, G., Dollár, P., Girshick, R.B., 2017\.  Mask R-CNN.  CoRR abs/1703.06870.  arXiv:1703.06870.
  * He et al. (2018) He, K., Gkioxari, G., Dollár, P., Girshick, R., 2018\.  Mask r-cnn.  arXiv:1703.06870.
  * He et al. (2015) He, K., Zhang, X., Ren, S., Sun, J., 2015.  Deep residual learning for image recognition.  arXiv:1512.03385.
  * Huang et al. (2020) Huang, Z., Wang, X., Wei, Y., Huang, L., Shi, H., Liu, W., Huang, T.S., 2020\.  Ccnet: Criss-cross attention for semantic segmentation.  arXiv:1811.11721.
  * Hurtik et al. (2020) Hurtik, P., Molek, V., Hula, J., Vajgl, M., Vlasanek, P., Nejezchleba, T., 2020\.  Poly-yolo: higher speed, more precise detection and instance segmentation for yolov3.  arXiv:2005.13243.
  * Isensee and Maier-Hein (2020) Isensee, F., Maier-Hein, K.H., 2020\.  Or-unet: an optimized robust residual u-net for instrument segmentation in endoscopic images.  arXiv:2004.12668.
  * Jha et al. (2021) Jha, D., Ali, S., Tomar, N.K., Riegler, M.A., Johansen, D., Johansen, H.D., Halvorsen, P., 2021\.  Exploring deep learning methods for real-time surgical instrument segmentation in laparoscopy, in: 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), pp. 1--4.  doi:10.1109/BHI50953.2021.9508610.
  * Kaul et al. (2019) Kaul, C., Manandhar, S., Pears, N., 2019.  Focusnet: An attention-based fully convolutional network for medical image segmentation, in: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 455--458.  doi:10.1109/ISBI.2019.8759477.
  * Lee and Park (2019) Lee, Y., Park, J., 2019.  Centermask : Real-time anchor-free instance segmentation.  CoRR abs/1911.06667.
  * Lin et al. (2017a) Lin, G., Milan, A., Shen, C., Reid, I., 2017a.  Refinenet: Multi-path refinement networks for high-resolution semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1925--1934.
  * Lin et al. (2017b) Lin, T., Goyal, P., Girshick, R.B., He, K., Dollár, P., 2017b.  Focal loss for dense object detection.  CoRR abs/1708.02002.
  * Lin et al. (2017c) Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S., 2017c.  Feature pyramid networks for object detection.  arXiv:1612.03144.
  * Liu et al. (2016) Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C., 2016.  Ssd: Single shot multibox detector.  Lecture Notes in Computer Science , 21–37doi:10.1007/978-3-319-46448-0_2.
  * Long et al. (2015a) Long, J., Shelhamer, E., Darrell, T., 2015a.  Fully convolutional networks for semantic segmentation, in: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431--3440.  doi:10.1109/CVPR.2015.7298965.
  * Long et al. (2015b) Long, J., Shelhamer, E., Darrell, T., 2015b.  Fully convolutional networks for semantic segmentation.  arXiv:1411.4038.
  * Maier-Hein et al. (2021a) Maier-Hein, L., Eisenmann, M., Sarikaya, D., März, K., Collins, T., Malpani, A., Fallert, J., Feussner, H., Giannarou, S., Mascagni, P., Nakawala, H., Park, A., Pugh, C., Stoyanov, D., Vedula, S.S., Cleary, K., Fichtinger, G., Forestier, G., Gibaud, B., Grantcharov, T., Hashizume, M., Heckmann-Nötzel, D., Kenngott, H.G., Kikinis, R., Mündermann, L., Navab, N., Onogur, S., Sznitman, R., Taylor, R.H., Tizabi, M.D., Wagner, M., Hager, G.D., Neumuth, T., Padoy, N., Collins, J., Gockel, I., Goedeke, J., Hashimoto, D.A., Joyeux, L., Lam, K., Leff, D.R., Madani, A., Marcus, H.J., Meireles, O., Seitel, A., Teber, D., Ückert, F., Müller-Stich, B.P., Jannin, P., Speidel, S., 2021a.  Surgical data science -- from concepts toward clinical translation.  arXiv:2011.02284.
  * Maier-Hein et al. (2021b) Maier-Hein, L., Wagner, M., Ross, T., Reinke, A., Bodenstedt, S., Full, P.M., Hempe, H., Mindroc-Filimon, D., Scholz, P., Tran, T.N., Bruno, P., Kisilenko, A., Müller, B., Davitashvili, T., Capek, M., Tizabi, M.D., Eisenmann, M., Adler, T.J., Gröhl, J., Schellenberg, M., Seidlitz, S., Lai, T.Y.E., Pekdemir, B., Roethlingshoefer, V., Both, F., Bittel, S., Mengler, M., Mündermann, L., Apitz, M., Kopp-Schneider, A., Speidel, S., Nickel, F., Probst, P., Kenngott, H.G., Müller-Stich, B.P., 2021b.  Heidelberg colorectal data set for surgical data science in the sensor operating room.  Scientific Data 8, 101\.  doi:10.1038/s41597-021-00882-2.
  * Milletari et al. (2018) Milletari, F., Rieke, N., Baust, M., Esposito, M., Navab, N., 2018.  Cfcm: Segmentation via coarse to fine context memory, in: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2018, Springer International Publishing, Cham. pp. 667--674.
  * Ni et al. (2019) Ni, Z., Bian, G., Xie, X., Hou, Z., Zhou, X., Zhou, Y., 2019.  Rasnet: Segmentation for tracking surgical instruments in surgical videos using refined attention segmentation network.  CoRR abs/1905.08663.
  * Ren et al. (2015) Ren, S., He, K., Girshick, R.B., Sun, J., 2015.  Faster R-CNN: towards real-time object detection with region proposal networks.  CoRR abs/1506.01497.
  * Ronneberger et al. (2015a) Ronneberger, O., Fischer, P., Brox, T., 2015a.  U-net: Convolutional networks for biomedical image segmentation, in: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (Eds.), Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015, Springer International Publishing, Cham. pp. 234--241.
  * Ronneberger et al. (2015b) Ronneberger, O., Fischer, P., Brox, T., 2015b.  U-net: Convolutional networks for biomedical image segmentation.  CoRR abs/1505.04597.  arXiv:1505.04597.
  * (39) Roß, T., Reinke, A., .  Robustmis2019.  URL: https://phabricator.mitk.org/source/rmis2019/.
  * Roß et al. (2021) Roß, T., Reinke, A., Full, P.M., Wagner, M., Kenngott, H., Apitz, M., Hempe, H., Mindroc-Filimon, D., Scholz, P., Tran, T.N., Bruno, P., Arbeláez, P., Bian, G.B., Bodenstedt, S., Bolmgren, J.L., Bravo-Sánchez, L., Chen, H.B., González, C., Guo, D., Halvorsen, P., Heng, P.A., Hosgor, E., Hou, Z.G., Isensee, F., Jha, D., Jiang, T., Jin, Y., Kirtac, K., Kletz, S., Leger, S., Li, Z., Maier-Hein, K.H., Ni, Z.L., Riegler, M.A., Schoeffmann, K., Shi, R., Speidel, S., Stenzel, M., Twick, I., Wang, G., Wang, J., Wang, L., Wang, L., Zhang, Y., Zhou, Y.J., Zhu, L., Wiesenfarth, M., Kopp-Schneider, A., Müller-Stich, B.P., Maier-Hein, L., 2021.  Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the robust-mis 2019 challenge.  Medical Image Analysis 70, 101920\.  doi:https://doi.org/10.1016/j.media.2020.101920.
  * Seo et al. (2018) Seo, P.H., Lin, Z., Cohen, S., Shen, X., Han, B., 2018\.  Progressive attention networks for visual attribute prediction.  arXiv:1606.02393.
  * Sheetz et al. (2020) Sheetz, K.H., Claflin, J., Dimick, J.B., 2020.  Trends in the Adoption of Robotic Surgery for Common Surgical Procedures.  JAMA Network Open 3, e1918911--e1918911.
  * Shvets et al. (2018) Shvets, A.A., Rakhlin, A., Kalinin, A.A., Iglovikov, V.I., 2018\.  Automatic instrument segmentation in robot-assisted surgery using deep learning, in: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 624--628.  doi:10.1109/ICMLA.2018.00100.
  * Sinha and Dolz (2021) Sinha, A., Dolz, J., 2021\.  Multi-scale self-guided attention for medical image segmentation.  IEEE Journal of Biomedical and Health Informatics 25, 121--130.  doi:10.1109/JBHI.2020.2986926.
  * Wang et al. (2020) Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., Liu, W., Xiao, B., 2020\.  Deep high-resolution representation learning for visual recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence , 1--1doi:10.1109/TPAMI.2020.2983686.
  * Wang et al. (2020) Wang, X., Zhang, R., Kong, T., Li, L., Shen, C., 2020\.  Solov2: Dynamic and fast instance segmentation.  arXiv:2003.10152.
  * Wang et al. (2018) Wang, Y., Deng, Z., Hu, X., Zhu, L., Yang, X., Xu, X., Heng, P.A., Ni, D., 2018.  Deep attentional features for prostate segmentation in ultrasound, in: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2018, Springer International Publishing, Cham. pp. 523--530.
  * Ward et al. (2021) Ward, T.M., Mascagni, P., Ban, Y., Rosman, G., Padoy, N., Meireles, O., Hashimoto, D.A., 2021.  Computer vision in surgery.  Surgery 169, 1253--1256.  doi:10.1016/j.surg.2020.10.039.
  * Wiesenfarth et al. (2021) Wiesenfarth, M., Reinke, A., Landman, B.A., Eisenmann, M., Saiz, L.A., Cardoso, M.J., Maier-Hein, L., Kopp-Schneider, A., 2021\.  Methods and open-source toolkit for analyzing and visualizing challenge results.  Scientific Reports 11, 1--15.
  * Woo et al. (2018) Woo, S., Park, J., Lee, J.Y., Kweon, I.S., 2018.  Cbam: Convolutional block attention module.  arXiv:1807.06521.
  * Xu et al. (2020) Xu, J., Wang, W., Wang, H., Guo, J., 2020.  Multi-model ensemble with rich spatial information for object detection.  Pattern Recognition 99, 107098\.  doi:https://doi.org/10.1016/j.patcog.2019.107098.
  * Xu et al. (2016) Xu, Y., Li, Y., Liu, M., Wang, Y., Lai, M., Chang, E.I.C., 2016.  Gland instance segmentation by deep multichannel side supervision.  arXiv:1607.03222.
  * You et al. (2016) You, Q., Jin, H., Wang, Z., Fang, C., Luo, J., 2016\.  Image captioning with semantic attention, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, Los Alamitos, CA, USA. pp. 4651--4659.  doi:10.1109/CVPR.2016.503.
  * Zhao et al. (2021) Zhao, Z., Jin, Y., Chen, J., Lu, B., Ng, C.F., Liu, Y.H., Dou, Q., Heng, P.A., 2021.  Anchor-guided online meta adaptation for fast one-shot instrument segmentation from robotic surgical videos.  Medical Image Analysis 74, 102240\.  doi:https://doi.org/10.1016/j.media.2021.102240.
  * Zlocha et al. (2019) Zlocha, M., Dou, Q., Glocker, B., 2019.  Improving retinanet for ct lesion detection with dense masks from weak recist labels.  arXiv:1906.02283.