# SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of
Surgical Tools

Luca Sestini<EMAIL_ADDRESS>Benoit Rosa Elena De Momi Giancarlo Ferrigno
Nicolas Padoy ICube, University of Strasbourg, CNRS, France  Department of
Electronics, Information and Bioengineering, Politecnico di Milano, Milano,
Italy  IHU Strasbourg, Strasbourg, France

###### Abstract

Instance segmentation of surgical instruments is a long-standing research
problem, crucial for the development of many applications for computer-
assisted surgery. This problem is commonly tackled via fully-supervised
training of deep learning models, requiring expensive pixel-level annotations
to train.
In this work, we develop a framework for instance segmentation not relying on
spatial annotations for training. Instead, our solution only requires binary
tool masks, obtainable using recent unsupervised approaches, and binary tool
presence labels, freely obtainable in robot-assisted surgery. Based on the
binary mask information, our solution learns to extract individual tool
instances from single frames, and to encode each instance into a compact
vector representation, capturing its semantic features. Such representations
guide the automatic selection of a tiny number of instances (8 only in our
experiments), displayed to a human operator for tool-type labelling. The
gathered information is finally used to match each training instance with a
binary tool presence label, providing an effective supervision signal to train
a tool instance classifier.
We validate our framework on the EndoVis 2017 and 2018 segmentation datasets.
We provide results using binary masks obtained either by manual annotation or
as predictions of an unsupervised binary segmentation model. The latter
solution yields an instance segmentation approach completely free from spatial
annotations, outperforming several state-of-the-art fully-supervised
segmentation approaches.

##  1 Introduction

Fig. 1: Examples of frame-wise and sequence-wise binary tool presence labels
for a robot-assisted surgery sequence from the EndoVis 2017 dataset (each
color represents a tool type). All the tools can be attached to the system at
the same time, while being visible only in certain frames.

Endoscopic videos from minimally-invasive procedures offer rich information
describing the surgical act. The automatic analysis of such information opens
up several opportunities to better understand surgical practice and to improve
it [10, 24, 28]. Surgical computer vision provides the necessary tools to
process raw endoscopic videos, enabling the extraction of dense information
for downstream applications. Among the various surgical computer vision tasks,
automatic instrument localisation and identification represent an essential
component of many downstream applications, like surgical skill assessment
[24], augmented reality [42], 3D scene reconstruction [46] and 3D pose
estimation [2]. This problem is often formalized by means of either semantic
or instance segmentation. Semantic Segmentation (SeS) aims at directly
labelling each image pixel as either belonging to the background class or to a
certain tool type class. Instance Segmentation (IS) aims at localising and
identifying individual tool instances, providing, for each instance, a
separate mask and a tool-type class labels. Such tool instantiation
information, i.e. the availability of a separate segmentation mask for each
tool instance present in the image, is extremely precious for downstream
applications like automatic skill assessment, as it enables individual tool
tracking over time. State-of-the-art approaches commonly tackle tool
segmentation via fully-supervised training of deep learning models [41, 17,
21, 22]. Such approaches require the availability of pixel-level semantic and
instance labels, extremely expensive to collect via manual annotation at a
large scale. This confines the training of such models to small annotated
datasets, limiting their generalization ability.
Recently, alternatives to standard fully-supervised approaches have been
proposed for the task of binary tool segmentation, a type of SeS featuring
only two classes, tool and background [38, 40, 33, 39, 8]. Most of these
solutions rely on semi-synthetic dataset generation, for example by combining
simulation data and domain translation approaches [38]. While appealing, their
application is still potentially limited by the domain gap between synthetic
and real data, and by the need for ad-hoc setups to collect synthetic data. A
few alternative works have shown the potential of prior instrument knowledge
to train deep learning models for binary segmentation, without requiring
spatial annotations. This has been done by exploiting prior knowledge on
instrument motion and shape [40], or, in the robotic context, by incorporating
3D tool models and kinematic data [33, 39, 8].
Despite these growing efforts to reduce the dependency on manual annotations,
research has remained confined to the binary segmentation task. We believe
that this is due to the rigid problem formalization imposed by common instance
and semantic segmentation approaches: such approaches do not benefit from the
potential availability of binary segmentation masks, as they would still
require pixel-level labels to train. Furthermore, this problem formalization
prevents the incorporation of significantly cheaper sources of semantic
information, compared to spatial annotations, like binary tool presence
labels. Specifically, we define as frame-wise these binary tool presence
labels describing which tool types are effectively visible in each frame; we
define as sequence-wise these labels indicating which tool types are
potentially visible in each frame (see Figure 1 for examples of a sequence).
While frame-wise labels are usually obtained via manual annotation - although
much cheaper than spatial annotation - sequence-wise labels can be
automatically obtained from different sources. In robot-assisted surgery, for
example, robotic systems can often record which tools are attached [22]. This
information only indicates that a certain tool could be visible at some point
while it is attached, but does not guarantee its visibility in any specific
frame (therefore a sequence-wise visibility). As a generalization, surgical
phase and step annotations could provide similar information, when a mapping
between phases/steps and tools can be approximately defined, for example by
knowing which tools are commonly used in each phase/step [31]. While the use
of frame-wise labels has been explored by weakly-supervised tool detection
methods [44, 30], no segmentation solutions have yet included them in their
training pipelines. Furthermore, to the best of our knowledge, no approaches
have yet explored the use of automatically obtainable sequence-wise labels.

Fig. 2: Overview of the proposed Spatial Annotation Free framework for
Instance Segmentation (SAF-IS). Top: training architecture highlighting the
three core steps. Tool instantiation is learnt from binary masks, potentially
obtained using recent unsupervised segmentation methods. Feature
representation learning is performed using a contrastive learning strategy,
powered by local temporal tracking. This step allows us to extract a feature
representation of each tool instance in the training set. Instance
classification is performed by incorporating a minimal amount of human-
provided information (prototype labels, as few as 8 in our experiments) and
cheaply obtainable binary tool presence labels. Bottom: SAF-IS inference
architecture.

In this work we propose a framework for instance segmentation model training,
which embraces the recent progress on unsupervised binary segmentation and the
availability of cheap binary tool presence labels, either frame-wise or
sequence-wise. Compared to pixel-level annotations, binary tool presence
labels are not spatially localized. Weakly-supervised tool detection
approaches [44, 30] exploit the class-activation maps provided by a classifier
trained on the frame-wise binary tool presence labels, in order to localize
the tools in the image space. However, such localization is commonly limited
to discriminant parts of the tools, like the tip, thus not suitable for
segmentation. In addition, such approaches cannot handle sequence-wise labels,
as these labels do not provide a ground truth signal for the training of the
frame-wise classifier. To tackle these challenges our solution first learns to
localize individual tool instances and to encode each of them in a compact
feature representation. These instance-wise representations are then used to
select a small number of tool instances (prototype instances, 8 only in our
experiments), which are presented to a human operator for tool-type labelling.
The gathered information is finally used to match each instance to a semantic
label from the corresponding set of binary tool presence labels, providing an
effective supervision signal for the training of an instance classifier.
To this aim, we make the following contributions:

  * •

we develop an unsupervised approach for tool instantiation (Figure 2, Tool
instantiation). This step allows training a model to extract a separate binary
segmentation mask for each tool instance present in a frame. With no
availability of pixel-level labels, we fabricate a pseudo-supervision signal
from the connected component instantiation of the binary masks, and refine it
using simple assumptions on instrument positioning in the image space. This
signal is used to train the instantiation model, directly predicting the
position of each instance centroid in the image space in the form of a 2D
displacement field;

  * •

we develop a self-supervised approach for feature representation learning
(Figure 2, Feature representation learning): this step allows training a model
to encode each tool instance in a compact representation, capturing its
semantic features. With no availability of pixel-level semantic labels, we
learn such representations by relying on intrinsic temporal information from
video sequences. Specifically, we design a contrastive learning approach based
on local instance tracking to draw positive and negative samples. This step
allows obtaining powerful instance-wise feature representations, providing the
necessary information to solve the final classification training step;

  * •

we develop an approach to learn instance classification from the binary tool
presence labels (Figure 2, Instance classification). The feature
representations of all the training instances, learnt at the previous step,
are used to guide the automatic selection of a tiny number of prototype
instances, displayed to a human operator for tool-type labelling. The gathered
information is propagated to the whole training set, allowing us to label each
training instance with a pseudo tool-type label (prototype labels). This
information is combined with the available binary tool presence labels (either
frame-wise or sequence-wise) using a teacher-student approach. This step
allows matching each training instance to a semantic label from the
corresponding set of binary tool presence labels, providing an effective
supervision signal for the training of the student instance classifier;

  * •

at inference time the trained architecture can perform instance segmentation
on single frames, by extracting individual tool instances, encoding each of
them in a compact feature representation, and separately classifying them
(Figure 2, bottom).

##  2 Related Work

Surgical instrument segmentation is a long-standing research problem. Before
the Deep Learning (DL) break-through, the problem was tackled by totally
relying on prior knowledge about surgical tools, like color distribution [47],
shape [6] or orientation in the field-of-view [45].
Following the DL irruption in the field, a great research effort has been
dedicated to designing powerful fully-supervised architectures, boosting
segmentation accuracy. Such solutions are presented below in Section 2.1,
focusing in particular on instance segmentation approaches. Although fully-
supervised methods have achieved unprecedented segmentation results on
benchmark datasets, their scalability is restricted by the need for manual
annotations, which confines their training to small annotated datasets,
limiting their generalization ability. To address this challenge, various
approaches have been suggested, which we present in Section 2.2.

###  2.1 Fully Supervised Solutions

Following the DL breakthrough in the field of surgical computer vision,
research works have mostly addressed the problem of surgical tool segmentation
using fully-supervised DL approaches. In particular, encoder-decoder
architectures based on Convolutional Neural Networks (CNNs) have been widely
adopted, in concurrency with a semantic segmentation formulation of the
problem. [11, 41, 32, 13] propose different variations of the U-Net
architecture [35], exploring different loss functions, residual connections,
dilated convolutions and ad-hoc augmentation pipelines. Multi-task learning
has also been adopted, coupling the segmentation task with image-based
localisation of tool landmarks [23] and task-oriented saliency maps prediction
[15]. While the segmentation task can be solved for single frames, temporal
information has been proven to boost performance, especially in the case of
partially occluded tools [17].
Recently, instance segmentation approaches have started gaining traction.
Several of the proposed approaches are based on the popular Mask-RCNN
architecture [14]. [21] directly train a Mask-RCNN architecture for the task
of surgical instrument instance segmentation. ISI-Net [12] adds a temporal-
consistency module for improved segmentation results. [22] propose a mask-
then-classify approach, adopting an anchor-free approach for instrument
instantiation, based on direct localisation of instruments centroids.
Differently from the above-listed methods, [52] simultaneously tackle the
problems of instance segmentation and tracking using a transformer
architecture based on the popular TrackFormer and DETR models [7, 29].
In this work we also adopt an instance segmentation problem formalization,
showing its benefits beyond fully-supervised training.

###  2.2 Non Fully-Supervised Solutions

Motivated by the need to reduce the burden of manual annotation, several
solutions have tackled the segmentation problem by including unlabelled data
in the training process, exploiting small sets of labelled data, weak
annotations or prior knowledge. Such solutions, mostly focusing on the binary
segmentation problem, are presented below.
Semi-Supervised solutions: this family of approaches incorporates unlabelled
data in the training process, while still requiring access to a set of
manually annotated data. Different solutions to combine unlabelled and
labelled data have been explored. [36] pre-train a CNN on unlabelled data, by
means of a pretext task carried out using a cycle-GAN architecture, and then
fine-tune the model on annotated data. A similar pipeline can be followed by
replacing the pre-text task with self-supervised representation learning on
the unlabelled data, as experimentally proven by [34]. [51] tackle the problem
of sparsely annotated data, propagating low hertz annotations to intermediate
unlabelled frames using optical-flow. [18] incorporate unlabelled data from
different domains in the training process to improve generalization to these
domains. This is achieved by mapping annotated frames from the labelled set to
the unlabelled domain using a cycle-GAN architecture, allowing for better
generalization.
Weakly-Supervised solutions: weakly-supervised training is a learning paradigm
trying to solve a task using annotations cheaper to obtain compared to the
ones required by the fully-supervised paradigm. Such annotations could be a
simplification of the ideal ground truth annotations, like scribbles in place
of masks for the segmentation task, or annotations providing
indirect/incomplete supervision for the targeted task (e.g. binary tool
presence labels for tool localisation tasks). For segmentation, the
application of weakly-supervised training remains confined to the binary task.
[25] propose a framework to integrate scribble-like annotations, speeding up
the annotation process. [50] automatically obtain a pseudo-supervision signal
by attaching an electromagnetic sensor to the surgical instruments. While
cutting the cost of annotations, the approach is inherently limited by
regulatory constraints, which limit the extent of validation of this study.
Weak annotations, in the form of frame-wise binary tool presence labels, have
mostly been used to tackle the problem of bounding-box localisation. [44]
train a multi-label classifier to predict tool presence from single frames;
the designed architecture features an extended spatial pooling layer yielding
class-specific feature maps, used during inference to localise the tools.
Similarly [30] use Wildcat Pooling [9] to obtain localisation maps, adding a
convolution-LSTM module for improved temporal consistency. Differently from
these two approaches, [49] use binary tool presence labels, in combination
with green-screen recorded images of surgical instruments, to obtain a pseudo-
supervision signal consisting of noisy and redundant bounding boxes. A
bounding-box regressor is then trained on the noisy supervision signal, and
its predictions for a certain tool are averaged together according to their
confidence score.
The use of frame-wise binary tool presence labels has remained limited to the
bounding-box detection task, as the standard approach involving using class-
activation maps limits the localisation to discriminative parts of the tools,
missing out significant parts of the instruments like the shafts. Furthermore,
research works on weakly-supervised learning have mostly focused on frame-wise
labels, which still require a certain annotation effort. This has led to
overlooking the opportunity given by sequence-wise binary tool presence
labels, particularly valuable, for example, in robot-assisted surgery.
Prior knowledge based solutions: as proven by early works on tool
segmentation, general assumptions on color distribution of endoscopic frames,
instrument position and prior shape knowledge, can be a sufficient source of
information to localise surgical instruments. [26], for example, generate
segmentation pseudo-labels using handcrafted cues, such as color distribution;
binary segmentation results are then refined exploiting feature correlation
between adjacent video frames. [40] propose FUN-SIS, an approach exploiting
general assumptions on instrument motion and shape-priors to train a binary
segmentation model, achieving results comparable to the ones of fully-
supervised solutions.
In this work we combine the use of prior knowledge and binary tool presence
labels to learn instance segmentation of surgical instruments. Prior knowledge
on instrument positioning in the field-of-view is exploited to instantiate
binary segmentation masks. Weak information, in the form of binary tool
presence labels, both frame-wise and sequence-wise, is then incorporated to
achieve accurate instance classification.

##  3 Methodology

The proposed SAF-IS framework for Spatial Annotation Free Instance
Segmentation explicitly separates the task into three core components: tool
instantiation, feature representation and instance classification. Differently
from standard semantic/instance segmentation approaches, SAF-IS does not
require spatial annotation of the training data. Instead, it relies on the
availability of binary segmentation masks, which can be cheaply obtained using
emerging unsupervised approaches, and binary tool presence labels.
The full framework is presented in Figure 2 and detailed below.

###  3.1 Tool Instantiation

Instrument instantiation is here defined as the problem of predicting, from an
endoscopic image $I$, the set of binary masks $\\{M^{Inst}_{i}\\}$, with $i$
in $[1,N_{Inst}]$, each one corresponding to an individual instrument visible
in the image. When the ground truth instantiation is known, the problem is
often formulated as bounding-box prediction [21, 12]. However, the
effectiveness of this approach has been questioned in [22], which proposed an
alternative solution based on direct regression of instance centroids’
position. We here adopt a similar formulation, showing its benefits with
respect to bounding-box prediction beyond fully-supervised learning.
The instantiation problem is here formalized as learning the mapping between
the image $I\in R^{W\times H\times 3}$ and the displacement field $D\in
R^{W\times H\times 2}$, uniquely assigning each tool pixel to an instance.
Given a pixel $\mathbf{p}=[p_{x},p_{y}]$, $D|_{\mathbf{p}}$ is equal to the
vector $\mathbf{v}=[c^{i}_{x}-p_{x},c^{i}_{y}-p_{y}]$ if $\mathbf{p}$ belongs
to a certain instance $i$, having its centroid in $[c^{i}_{x},c^{i}_{y}]$, or
to the null vector $[0,0]$, if $\mathbf{p}$ belongs to the background. Given a
training set with known ground truth instantiation $D$, such mapping can be
learnt by an instantiation model, implemented as a neural network, by using a
fully-supervised training formulation, as in [22]. This can be achieved by
optimizing the loss $L^{FS}_{I}$, implemented as the pixel-wise distance
between the ground truth displacement field $D$ and the instantiation model
prediction $\tilde{D}$:

$L^{FS}_{I}=|D-\tilde{D}|.$ (1)

At inference time, given a new image $I$ and the corresponding predicted
displacement field $\tilde{D}$, the set of instance masks $\\{M^{Inst}_{i}\\}$
can be easily extracted by identifying the instance centroids, as the pixels
where the displacement field converges, and assigning each tool pixel to the
centroid pointed by the corresponding displacement vector.

Fig. 3: Overview of the proposed strategy to generate a pseudo-supervision
signal to learn instrument instantiation. Given an image $I$, its binary mask
$M^{B}$ is instantiated using a Connected Component (CC) algorithm, yielding
the set of tool masks $\\{M^{CC}_{i}\\}$, with $i$ in $[1,N_{CC}]$. From them,
the displacement field $D^{CC}$ and the overlap mask $M^{OV}$ can be
automatically obtained. A random tool instance is then selected from the
training set, and pasted on $I$, $M^{B}$, $D^{CC}$, producing their augmented
versions $I^{\mathbf{*}}$, $M^{B\mathbf{*}}$, $D^{CC\mathbf{*}}$.

Fig. 4: Overview of the proposed instantiation strategy. Given an image $I$
the trained instantiation model predicts the masked displacement field
$\tilde{D}$. A square grid is then overlapped to $\tilde{D}$, and the squares
with high convergence (per-pixel average $>\varepsilon_{C}$) are then
extracted, and separated by Connected Component (CC) labelling, yielding a set
of $\tilde{N}_{Inst}$ centroid regions. Each tool pixel is then assigned to
the corresponding centroid, yielding the set of instance masks
$\\{\tilde{M}^{Inst}_{i}\\}$, with $i$ in $[1,\tilde{N}_{Inst}]$.

Training: In our case, only the binary mask $M^{B}$ is known. Without a ground
truth instantiation, we rely on the assumption that surgeons tend to avoid
overlapping surgical instruments in the field-of-view, in order to reduce the
chances of mutual tool occlusions and unwanted tool interactions.
Given an image $I$ and the corresponding binary mask $M^{B}$, if tools do not
overlap, the instance masks can be obtained by separating the Connected
Components (CC) of $M^{B}$ through standard computer vision methods like the
Spaghetti algorithm [5]. The displacement field $D^{CC}$, approximating the
ground truth $D$, can then be directly obtained from the set of $N_{CC}$ tool
masks $\\{M^{CC}_{i}\\}$, with $i$ in $[1,N_{CC}]$, by subtracting each tool
pixel position from the centroid $[c^{i}_{x},c^{i}_{y}]$ of the corresponding
mask $M^{CC}_{i}$. While effective in the case of non-overlapping tools, CC
labelling systematically fails when tools overlap. In order to mitigate this
problem we artificially modify the supervision signal obtained from CC
instantiation, as follows:

  * •

potential overlapping tools identification: in minimally invasive surgery
surgeons adopt the principle of triangulation to increase their ability to
visualize and access anatomy [37]. As a result, surgical tools commonly enter
the camera’s field-of-view from the sides. Therefore, given the set of CC
masks $\\{M^{CC}_{i}\\}$, $M^{CC}_{i}$ is considered a potential overlapping
instance if it covers the full horizontal length of the frame (see Figure 3
for an example). All pixels corresponding to potential overlapping instances
are collected in the binary overlap mask $M^{OV}$, and discarded from loss
computation as described later in this Section;

  * •

instance pasting augmentation (AugmPaste): given an image $I$, its binary mask
$M^{B}$ and its CC displacement field $D^{CC}$, a random tool instance is
selected from a different training sample and pasted on them, yielding the
augmented image $I^{\mathbf{*}}$, the augmented binary mask $M^{B\mathbf{*}}$
and the augmented displacement field $D^{CC\mathbf{*}}$ (Figure 3). This
augmentation step allows us to artificially simulate the presence of
overlapping instances, making up for the discarded instances at the previous
step.

Given the image $I^{*}$, in addition to the displacement field $\tilde{D}$, we
let the instantiation model predict the binary segmentation mask
$\tilde{M}^{B}$, which we multiply by $\tilde{D}$ to ensure that the
displacement vector for pixels belonging to the background is a null vector
$[0,0]$. For simplicity, we keep the notation $\tilde{D}$ to refer to the
result of such product.
Given the image $I^{\mathbf{*}}$, the corresponding network predictions
$\tilde{D}$ and $\tilde{M}^{B}$, the binary mask $M^{B\mathbf{*}}$, the
displacement field $D^{CC\mathbf{*}}$ and the overlap mask $M^{OV}$, the
instantiation model is trained by optimizing the loss $L_{I}$:

$L_{I}=|D^{CC\mathbf{*}}-\tilde{D}|(1-M^{OV})+L_{CE}(M^{B\mathbf{*}},\tilde{M}^{B}),$
(2)

where $L_{CE}$ is a standard pixel-wise cross-entropy loss.

Fig. 5: Overview of the tracking strategy used to generate positive samples
for contrastive learning. Given two consecutive frames, the centroids at time
$t$, obtained from the displacement field $D_{t}$ are mapped to the $I_{t+1}$
space using optical flow $OF$, computed between the two images $I_{t}$ and
$I_{t+1}$. The projected centroids are then matched to the ones obtained from
the displacement field $D_{t+1}$. This allows building the set of tubes
$\\{T_{i}\\}$, with $i$ in $[1,\tilde{N}_{Inst}]$. Tubes are progressively
grown by repeating this process for consecutive frames.

Inference: given an image $I$ and the trained instantiation model, the
predicted displacement field $\tilde{D}$ must me mapped to the set of instance
masks $\\{\tilde{M}^{Inst}_{i}\\}$, with $i$ in $[1,\tilde{N}_{Inst}]$, and
$\tilde{N}_{Inst}$ being the number of predicted instances in a frame. While
for the ground truth displacement field $D$ each tool pixel vector points
exactly to the corresponding centroid pixel, this is not guaranteed for the
predicted $\tilde{D}$. Therefore we define as centroids the regions of
$\tilde{D}$ with a high rate of displacement vectors convergence. Practically,
we overlap a square grid to $\tilde{D}$ and compute, for each square, the per-
pixel average number of vectors pointing inside it. If such number is above a
predefined threshold $\varepsilon_{C}$, the square is considered a centroid
square. Connected squares are grouped together, to yield the set of centroid
regions $\\{c_{i}\\}$, with $i$ in $[1,\tilde{N}_{Inst}]$. The instance masks
can then be extracted by assigning each tool pixel $\mathbf{p}$ to the
centroid $c_{i}$ closest to the point identified by
$\mathbf{p}+\tilde{D}|_{\mathbf{p}}$. This yields the set of predicted
instance masks $\\{\tilde{M}^{Inst}_{i}\\}$, with $i$ in
$[1,\tilde{N}_{Inst}]$ (Figure 4). In our framework the predicted instance
masks are subsequently used to learn instance-wise feature representations, as
now discussed.

Fig. 6: Left: visualization of learnt feature representations of the EndoVis
2017 [3] training set instances, clustered ($N_{km}$ equal to 8) and projected
in the 2D space using t-SNE algorithm [27]. Each instance point is colored
with a different shade of grey to represent the cluster id. Prototype instance
features are marked with $\star$, and the corresponding masks are overlaid on
the frame and highlighted by a bounding-box, to facilitate their labelling by
a human operator. The color of the mask overlays represents the ground truth
tool type that the user would assign. In practice, this annotation step can be
carried out in 8 mouse clicks only. Right: prototype instance labels
propagation to the training set. Each instance-wise feature projection is
colored accordingly to its prototype label, assigned via propagation from the
prototype instance.

###  3.2 Feature Representation Learning

In the absence of pixel-level semantic labels, we rely on self-supervision to
learn robust and meaningful feature representations of each tool instance,
tailored for the instance segmentation task. The problem of self-supervised
representation learning has been often addressed by means of contrastive
learning in literature [16]. While general contrastive learning approaches
usually learn global frame-level feature representations, we find this
formulation to be ill-posed for the instrument segmentation problem, as it
lacks the spatial granularity necessary to discriminate between different
instances. Therefore we design an instance-level contrastive learning
approach, exploiting the unsupervised instantiation described above and
intrinsic temporal information from video sequences.
Given an image $I$ and the set of instance masks $\\{\tilde{M}^{Inst}_{i}\\}$
predicted by the instantiation model, we want to map each instance to a
feature vector $F_{i}$, capturing its semantic content. We obtain feature
vectors using a feature extractor model implemented using a standard ResNet-50
architecture. Specifically, for each instance, we pass $I$ through the model
and multiply the intermediate feature maps by ${\tilde{M}^{Inst}_{i}}$,
resized to match their dimensions, to obtain the corresponding instance-wise
feature vector $F_{i}$. Then, given a feature representation $F_{i}$,
intrinsic temporal information from the video sequence is used to draw
positive and negative examples for contrastive loss computation. Specifically:

  * •

positive examples $\\{F_{i}^{+}\\}$ are sampled from the instance tube
$T_{i}$, built from the frame-by-frame tracking of the instance $i$. Such
tracking is described in Figure 5. Given the consecutive images $I_{t}$ and
$I_{t+1}$, and their corresponding sets of instrument instances, tracking is
solved by projecting the centroids of $I_{t}$ into $I_{t+1}$ space using the
optical flow $OF$, computed between $I_{t}$ and $I_{t+1}$. Each $I_{t}$
centroid is then matched to the closest $I_{t+1}$ centroid. Optical flow
projection allows us to robustly handle tool movements between consecutive
frames, reducing the chances of wrong matching;

  * •

negative examples $\\{F_{i}^{-}\\}$ can be sampled either from different tubes
belonging to the same frame or from tubes far apart in time.

The feature extractor network is then trained by optimizing the loss $L_{F}$
between $\\{F_{i}^{+}\\}$ and $\\{F_{i}^{-}\\}$:

$L_{F}=L_{SCL}(\\{F_{i}^{+}\\},\\{F_{i}^{-}\\}),$ (3)

where $L_{SCL}$ is the Supervised Contrastive Loss formulation proposed in
[19], with each instance tube treated as a separate class. The learnt feature
representations are exploited in the next step for classifier training.

Fig. 7: Overview of the proposed weakly-supervised instance classification
module. Given an image $I$, the corresponding set of instance-wise features
$\\{F_{i}\\}$, with $i$ in $[1,\tilde{N}_{Inst}]$, is obtained from the
instance masks $\tilde{M}^{Inst}_{i}$. Each feature is mapped to the
corresponding prototype label $S^{P}_{i}$, which, as shown in this case, does
not necessarily correspond to the ground truth label. Each feature is also
independently passed through the Teacher (T) and Student networks (S),
yielding the predicted probabilities $\mathbf{\tilde{P}^{T}_{i}}$,
$\mathbf{\tilde{P}^{S}_{i}}$, and the corresponding predicted labels
$\tilde{S}^{T}_{i}$, $\tilde{S}^{S}_{i}$ (for the sake of readability only the
latter are shown in the picture). T is trained optimizing the loss $L_{T}$
computed using the prototype labels $\\{S^{P}_{i}\\}$. Simultaneously, T
predictions are used to compute the assignment costs $C_{\langle
i_{C},i_{P}\rangle}$ for each $i_{P}$ permutation of each $i_{C}$ combination
of the weak labels $\\{S^{W}_{i}\\}$, with $i$ in $[1,N_{W}]$. The ordered set
$[\bar{S}^{W}_{i}]$, with $i$ in $[1,\tilde{N}_{Inst}]$, corresponding to the
minimum assignment cost, is used to compute the loss $L_{S}$ for Student
network optimization.

###  3.3 Instance classification

Given the set of available tool type classes $\\{S_{i}\\}$, with $i$ in
$[1,N_{cls}]$, a classifier model must now be trained to learn the mapping
between instance features and class labels from that set. In the absence of
pixel-level semantic labels, we rely on binary tool presence labels to solve
this task, cheaper to collect via manual annotation, or even automatically
obtainable (e.g. sequence-wise labels from robotic systems). As binary tool
presence labels are not spatially localised, the matching between training
tool instances and binary tool presence labels must be defined. The class-
activation approach, commonly adopted for weakly-supervised object detection,
requires frame-wise ground-truth annotations about tool presence, which makes
it inapplicable to sequence-wise labels. We therefore propose a more flexible
solution, applicable to both frame-wise and sequence-wise labels. Our solution
is designed to solve the matching problem by injecting a minimal amount of
human knowledge, specifically collected to maximize its information content
while minimizing the annotation effort. Specifically, we automatically select
a tiny number of highly representative instances (protoype instances) and ask
a human operator to label them. The gathered information is then used to match
binary tool presence labels and instances, providing an effective supervision
signal for classifier training. The two steps are now detailed.
Prototype labelling: given the complete set of learnt features for all the
instances in the training set, unsupervised clustering is applied. In our
experiments we make use of the standard K-Means++ clustering algorithm [4],
with the number of clusters $N_{km}$ regarded as an hyper-parameter. The
$N_{km}$ instances corresponding to the clusters’ centroids are defined as
prototype instances. A human operator would now be required to assign a label
$S^{P}$ from the set $\\{S_{i}\\}$ to each prototype instance. In order to
propagate the prototype instance labels to the rest of the training instances,
we require all instances belonging to the same cluster to share the same
semantic label $S^{P}$. Figure 6 provides a visualization the of prototype
instance labelling process, and of the result of prototype label propagation.
In principle, a number of clusters $N_{km}$ equal to $N_{cls}$, the total
number of tool type classes available, is sufficient to correctly label the
whole training dataset, and potentially to directly deploy the instance
segmentation model: given an unseen image $I$ and a predicted tool instance
mask $\tilde{M}^{Inst}_{i}$ from that image, inference would then be performed
by extracting the corresponding feature vector $F_{i}$ and associating it to
the prototype label $S^{P}$ of the cluster closest to $F_{i}$ in the feature
space. However, in practice, as the feature learning step is imperfect, the
prototype labels can be noisy, as experimentally shown in Section 5.
Nonetheless, we show that the information provided by prototype labels can be
used to match binary tool presence labels and instances, providing an
effective supervision signal for classifier training.

Binary tool presence labels incorporation: let us consider the set of binary
tool presence labels $\\{S^{W}_{i}\\}$ with $i$ in $[1,N_{W}]$, subset of the
set of tool-type labels $\\{S_{i}\\}$, associated to a certain frame. As
discussed in Section 1, this information can be defined as frame-wise, if the
labels indicate which tool types are effectively visible in the frame, or
sequence-wise, if they indicate which tool types are visible at some point in
the sequence the frame belongs to, but not necessarily in such frame. Binary
tool presence information, either frame-wise or sequence-wise, does not
provide tool localisation information, and is therefore defined as weak with
respect to the segmentation task. While cheaply obtainable, such weak labels
are often overlooked by segmentation approaches, as they pose several
challenges:

  * •

differently from pixel-level labels, binary tool presence labels are not
directly matched to a specific instance, making them hard to digest for
standard segmentation architectures, designed to learn from pixel-level
annotations;

  * •

depending on the system/annotation protocol used to collect the information,
the presence of multiple instances of the same tool type may not be recorded.
In the Cholec80 dataset [43], for example, frame-wise binary tool presence
labels do not keep track of multiple tool instances;

  * •

sequence-wise labels commonly do not reflect which tool types are effectively
visible in each frame. In the case of robotic surgery, for example, tools are
attached beforehand to the robotic system, potentially remaining unused for
relatively long periods of time. Similarly for surgical phases, certain tools,
like the ones used for coagulation, may be linked to every phase of a
procedure, while being visible only for small amounts of time.

In order to make effective use of such information, each tool instance in a
frame must be matched to a weak label from the set $\\{S^{W}_{i}\\}$
associated to that frame. Once the matching is found, a classifier model can
be trained on the matched labels. In practice, the binary tool presence labels
softly constrain the training of the classifier, providing a reduced set of
tool-type labels among which the ground truth one for each instance is to be
found.
Let us consider an image $I$, the sets of instance masks, features and
prototype labels $\\{\tilde{M}^{Inst}_{i}\\},\\{F_{i}\\},\\{S^{P}_{i}\\}$,
with $i$ in $[1,\tilde{N}_{Inst}]$, and the set of weak labels
$\\{S^{W}_{i}\\}$, with $i$ in $[1,N_{W}]$ associated to $I$. Mining such weak
labels requires to find the function $\xi$, matching the set of
$\tilde{N}_{Inst}$ features to the set of $N_{W}$ weak labels. However, in the
most general case, such transformation is:

  * •

non injective, as there could be multiple instances sharing the same tool
label $S^{W}_{i}$;

  * •

non surjective, as a certain tool label $S^{W}_{i}$ may not be present in a
specific frame.

This implies that given the set of $\tilde{N}_{Inst}$ tool instances in a
frame, different combinations of $\tilde{N}_{Inst}$ elements of the $N_{W}$
weak labels are plausible. To simplify the problem, and avoid degenerate
solutions, we assume that if the number of instances $\tilde{N}_{Inst}$ in a
frame is equal or smaller than the number of weak labels for that frame, every
instance is assigned to a different label. Specifically, we identify the set
of plausible weak labels combinations as follows:

  * •

if $\tilde{N}_{Inst}<N_{W}$, all the possible combinations of
$\tilde{N}_{Inst}$ elements of the $N_{W}$ labels are plausible;

  * •

if $\tilde{N}_{Inst}==N_{W}$, we assume that the set of $N_{W}$ labels is the
only plausible combination;

  * •

if $\tilde{N}_{Inst}>N_{W}$, all the possible combinations with repetitions of
$\tilde{N}_{Inst}$ elements of the $N_{W}$ labels are plausible.

Among the set of plausible weak labels combinations, the correct label
combination must be identified, and the matching between each instance and
each weak label in such combination must be determined. This could be achieved
by associating to each $i_{P}$ permutation of each $i_{C}$ plausible
combination of the weak labels an assignment cost $C_{\langle
i_{C},i_{P}\rangle}$. Each couple $\langle i_{C},i_{P}\rangle$ yields an
ordered set of weak labels $[S^{W\langle i_{P},i_{C}\rangle}_{i}]$, with $i$
in $[1,\tilde{N}_{Inst}]$. Among them, the ordered set minimizing the
assignment cost could be selected and used for the classifier training.
To solve this problem we propose a teacher-student approach (Figure 7),
exploiting the knowledge gathered from the prototype labels. Teacher and
Student are two identical classifiers that map a feature vector $F_{i}$ to the
vectors $\mathbf{\tilde{P}^{T}_{i}}$, $\mathbf{\tilde{P}^{S}_{i}}$,
respectively. $\mathbf{\tilde{P}^{T}_{i}}$, $\mathbf{\tilde{P}^{S}_{i}}$
represent the predicted probability of the instance to belong to each of the
$N_{cls}$ classes, according to Teacher and Student, respectively. From
$\mathbf{\tilde{P}^{T}_{i}}$, $\mathbf{\tilde{P}^{S}_{i}}$ the class with the
highest probability $\tilde{S}^{T}_{i}$, $\tilde{S}^{S}_{i}$ is regarded as
the predicted label. The Teacher network is trained to map each feature
$F_{i}$ to the corresponding prototype label $S^{P}_{i}$, by optimizing the
instance classification loss $L_{T_{i}}$:

$L_{T_{i}}=L_{CE}(\mathbf{\tilde{P}^{T}_{i}},S^{P}_{i}).$ (4)

For each couple $\langle i_{C},i_{P}\rangle$, its assignment cost $C_{\langle
i_{C},i_{P}\rangle}$ can then be computed as the average cross-entropy loss
between the predicted probabilities $[\mathbf{\tilde{P}^{T}_{i}}]$ and the
weak labels $[S^{W\langle i_{P},i_{C}\rangle}_{i}]$, corresponding to that
couple, as follows:

$C_{\langle
i_{C},i_{P}\rangle}=\frac{1}{\tilde{N}_{Inst}}\sum_{i=1}^{\tilde{N}_{Inst}}L_{CE}(\mathbf{\tilde{P}^{T}_{i}},S^{W\langle
i_{C},i_{P}\rangle}_{i}).$ (5)

The ordered set of weak labels $[\bar{S}^{W,i_{P},i_{C}}_{i}]$, corresponding
to the couple $\langle i_{C},i_{P}\rangle$ minimizing the assignment cost, is
selected. The Student network is then trained by optimizing the instance
classification loss $L_{S_{i}}$, between the predicted probabilities
$[\mathbf{\tilde{P}^{S}_{i}}]$ and the matched weak labels
$[\bar{S}^{W,i_{P},i_{C}}_{i}]$:

$L_{S_{i}}=L_{CE}(\mathbf{\tilde{P}^{S}_{i}},\bar{S}^{W\langle
i_{C},i_{P}\rangle}_{i}).$ (6)

In practice, the Teacher network applies the knowledge gathered from the
prototype labels to identify the correct ordered set of weak labels used for
Student training. Doing so, the Teacher approximates the function $\xi$,
matching each of the $\tilde{N}_{Inst}$ tool instances to a weak label from
the set $\\{S^{W}_{i}\\}$.
This general framework applies to both frame-wise and sequence-wise binary
tool presence labels. In the case of frame-wise labels, $\xi$ becomes
surjective, significantly reducing the space of possible solutions and
facilitating the matching.

X

##  4 Experimental Set-up

The proposed framework was validated on the MICCAI 2017 and 2018 EndoVis
Robotic Instrument Segmentation Challenge datasets. The two datasets are now
introduced (Section 4.1), together with the specific design choices and
training details (Section 4.2).

###  4.1 Datasets

EndoVis2017 [3]: the original challenge dataset consists of 10 video clips,
resampled at a frame rate of 1 frame-per-second, of abdominal porcine
procedures, performed using the da Vinci robotic system. Each clip contains
300 high-resolution frames (1024 $\times$ 1280). During the challenge 8x225
frames were released for training, while the remaining 8x75 frames and two
additional clips were held out by the organizers for testing. A total of 7
tool classes are present in the dataset. We provide results on this dataset
according to the same evaluation protocol as [41], by performing 4-fold cross-
validation on the 8x225 released training data (regrouped in 4 splits). We
report the average metric over the 4 splits, for direct comparison with state-
of-the-art approaches.
EndoVis2018 [1]: the original challenge dataset contains 19 video clips,
resampled at a frame rate of 1 frame-per-second, of abdominal porcine
procedures, performed using da Vinci robotic system. Each video contains a
total of 300 high-resolution frames (1024 $\times$ 1280). During the challenge
15 clips were released for training, while the remaining clips were held out
by the organizers for testing. The dataset was originally annotated for
anatomy and tool-part segmentation, and did not feature instrument type
labels. [12] annotated with pixel-level semantic labels 149 frames for each of
the 15 training clips, and split them into a training set consisting of 11
clips, and a validation set containing the remaining 4 clips. The same 7 tool
classes from EndoVis2017 dataset were used. We provide results on this dataset
according to the same evaluation protocol as [12], by training on the 11
training clips, and validating on the remaining 4 clips.

As the proposed SAF-IS approach requires binary instrument masks to train, we
provide results using both manually annotated binary masks and automatically
segmented masks generated using the unsupervised FUN-SIS approach [40]. The
mean binary IoU for the FUN-SIS approach on the EndoVis2017 and EndoVis2018
datasets is equal to 83.7% and 81.3%, respectively.
Frame-wise binary tool presence labels were automatically generated for each
frame as the unique pixel-level semantic labels present in the corresponding
ground truth semantic masks. Sequence-wise binary tool presence labels were
also automatically generated, by considering each video clip in the datasets
as a sequence, and assigning to each clip, as sequence-wise labels, the full
set of unique semantic labels present in the ground truth semantic masks of
all the frames in the clip. For 46.12% of the frames in the EndoVis2018
dataset the sequence-wise labels do not correspond to the frame-wise labels
(40.72% for EndoVis2017 dataset), i.e., for a certain frame, its sequence-wise
labels contain at least a tool type which is not visible in it (but which is
present at some point in the clip it belongs to).

Superv. | Method | EndoVis
---|---|---
2017 | 2018
<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
GT | MRCNN | 76.11 | 61.87 | 75.01 | 63.12
SAF-IS | 88.40 | 72.12 | 78.57 | 66.00
CCM | MRCNN | 71.26 | 55.98 | 73.99 | 60.04
SAF-IS | 85.36 | 63.70 | 75.92 | 61.08
CCF | MRCNN | 63.81 | 44.99 | 62.48 | 42.31
SAF-IS | 81.31 | 56.14 | 71.01 | 49.17
Table 1: Tool instantiation results for the proposed SAF-IS approach and Mask-
RCNN on EndoVis 2017 and 2018 datasets, trained according to three modalities:
fully-supervised (GT) and unsupervised using Connected Component labelling of
manually annotated masks (CCM) and FUN-SIS predicted masks (CCF).

###  4.2 Design Choices & Training Details

Tool instantiation: the instantiation model is implemented as a U-Net
architecture with SegFormer encoder [48], available from the Segmentation
Models library in PyTorch. Training was carried out for 60 epochs using the
Adam optimizer with a learning rate equal to 1e-3 and a batch size of 32,
applying standard photometric and geometric augmentations from the
Albumentation library to the original images, resized to a $256\times 256$
resolution. During inference, centroids were selected by overlapping the
predicted displacement field with a square grid of $32\times 32$ resolution
(i.e. each grid square of $8\times 8$ pixel dimension); a threshold
$\varepsilon_{C}$ of 5 was used to select centroid squares (i.e. squares with
a per-pixel average of at least 5 displacement vectors pointing at them were
selected as centroids). The impact of grid resolution and threshold value is
investigated in Section 6.
Feature representation learning: the feature extractor network is implemented
as a ResNet-50 architecture. Each instance mask is multiplied by the output of
the $conv3\\_4$ layer. Instance-wise features are obtained by applying a
global average pooling to the output of the $conv5\\_3$ layer, having 2048
feature channels. Training was carried out for 80 epochs using the Adam
optimizer with a learning rate equal to 5e-5 and a batch size of 64, applying
standard photometric and geometric augmentations to the original images,
resized to a $512\times 512$ resolution. For the contrastive loss $L_{SCL}$ a
temperature factor equal to 0.1 was used.
Instance classification: for the main experiments (Section 5.2), K-Means++
clustering algorithm was applied with a total number of clusters $N_{km}$
equal to 8 (therefore 8 instances were required to be labelled by a human
user). While in the real scenario such assignment would be performed by a
human operator, as discussed in Section 6, it was here automatically performed
by associating to each prototype instance the semantic label of the ground
truth instance of the same frame having the maximum overlap according to the
Intersection-over-Union metric.
The classification networks (Teacher, Student) were implemented as a 2-layer
fully-connected network, with intermediate feature size of 512 and batch
normalization. Training was carried out for 40 epochs using the Adam optimizer
with a learning rate equal to 1e-4 and a batch size of 128, applying standard
photometric and geometric augmentations to the original images, resized to a
$512\times 512$ resolution.

##  5 Experiments and Results Analysis

We now present the experimental validation of the proposed SAF-IS framework,
and compare it with state-of-the-art approaches. Tool instantiation results
and complete instance segmentation results are separately presented in
Sections 5.1 & 5.2, respectively.

] table

Method | Supervision Type | EndoVis
---|---|---
Pixel-level | Weak | 2017 | 2018
S | I | B | P | FW | SW
Ternaus[41] | ✓ |  |  |  |  |  | 35.27 | /
MF-TN†[17] | ✓ |  |  |  |  |  | 37.35 | /
DMF-TN†[51] | ✓30% |  |  |  |  |  | 45.83 | /
DMF-TN†[51] | ✓20% |  |  |  |  |  | 43.71 | /
DMF-TN†[51] | ✓10% |  |  |  |  |  | 33.64 | /
M&C†‡[22] | ✓ | ✓ |  |  |  |  | 65.70 | /
ISI-Net†[12] | ✓ | ✓ |  |  |  |  | 55.62 | 73.03
MRCNN[21] | ✓ | ✓ |  |  |  |  | 42.28 | /
Tra-SeTr†[52] | ✓ | ✓ |  |  |  |  | 60.04 | 76.20
SAF-IS |  |  | ✓ | ✓0.3% |  |  | 43.86 | 56.62
SAF-IS |  |  | ✓ | ✓0.3% | ✓ |  | 53.73 | 63.38
SAF-IS |  |  | ✓ | ✓0.3% |  | ✓ | 52.64 | 63.57
SAF-IS |  |  |  | ✓0.3% |  |  | 30.47 | 54.08
SAF-IS |  |  |  | ✓0.3% | ✓ |  | 45.86 | 58.03
SAF-IS |  |  |  | ✓0.3% |  | ✓ | 42.41 | 57.75
Table 2: Instance segmentation results for the proposed SAF-IS approach,
state-of-the-art methods on EndoVis 2017 and 2018 datasets. Supervision
signals used by each approach are reported: pixel-level semantic labels (S,
percentage of labelled data reported for semi-supervised approaches), pixel-
level instance labels (I), required by fully-supervised instance segmentation
approaches, binary segmentation masks (B, for SAF-IS, if not checked FUN-SIS
predicted masks are used), prototype labels (P, 8 labels in total in these
experiments, $\sim$0.3% of total training instances), frame-wise tool presence
labels (FW) and sequence-wise tool presence labels (SW). † methods using
temporal information at inference time. ‡ methods using additional tool-part
annotations for training.

Augm. | EndoVis
---|---
OV | PS | 2017 | 2018
<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
|  | 74.85 | 56.585 | 71.56 | 58.08
✓ |  | 77.74 | 54.82 | 77.58 | 62.00
| ✓ | 81.91 | 59.82 | 70.54 | 57.98
✓ | ✓ | 85.35 | 63.70 | 75.92 | 62.08
(a)

Augm. | EndoVis
---|---
OV | PS | 2017 | 2018
<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
|  | 67.82 | 47.86 | 65.23 | 43.94
✓ |  | 71.80 | 45.69 | 71.91 | 48.99
| ✓ | 72.41 | 49.42 | 67.99 | 47.12
✓ | ✓ | 81.31 | 56.14 | 71.01 | 49.16
(b)

Table 3: Results of the ablation study on unsupervised instrument
instantiation from manually annotated (a) and FUN-SIS predicted (b) binary
masks, highlighting the separate and combined impact of: masking of
potentially overlapping instances (OV) and pasting of random tool instances
(PS).

###  5.1 Tool Instantiation

In order to analyze tool instantiation quality, we evaluate results according
to a class-agnostic Average-Precision metric, computed for two values of
threshold Intersection-Over-Union (IoU<EMAIL_ADDRESS>(50%<EMAIL_ADDRESS>(70%). We
present results obtained by our unsupervised approach using, as binary masks,
both manual annotations (SAF-IS CCM) and unsupervised FUN-SIS predictions
(SAF-IS CCF). In addition, we report results for the instantiation model
trained in a fully-supervised manner on the ground truth displacement field
(SAF-IS GT). As, to the best of our knowledge, no other work has previously
attempted unsupervised instantiation of binary tool masks, we compare our
solution against a Mask-RCNN baseline, trained under the same fully-supervised
(MRCNN GT) and unsupervised modalities (MRCNN CCM, MRCNN CCF). However, as
Mask-RCNN is an anchor-based approach, the local masking for automatically
identified overlapping tools ($M^{OV}$, described in Section 3.1), is not
easily implementable, and would require substantial architectural
modifications which are beyond the scope of this work. Therefore we limit the
augmentation strategy for unsupervised Mask-RCNN experiments to instance
pasting, described in Section 3.1.
Results presented in Table 1 show how our proposed solution outperforms Mask-
RCNN across both datasets and for all three training modalities. A similar
result for the fully-supervised training modality was already presented in
[22]. These experiments highlight the benefits of tool instantiation based on
direct centroid regression, beyond full-supervision, for the unsupervised
setting. Indeed, the unsupervised SAF-IS solution using binary annotated masks
(SAF-IS CCM) closely follows the fully-supervised one (SAF-IS GT), with an
average gap of -$\Delta$3.3%<EMAIL_ADDRESS>across the two datasets. In addition, the
greatest performance gap between SAF-IS and Mask-RCNN is found when using FUN-
SIS binary masks to train (CCF): +$\Delta$17.5%<EMAIL_ADDRESS>and +$\Delta$11.15%
<EMAIL_ADDRESS>in the EndoVis 2017 dataset. This result shows how our solution is
particularly suitable to handle a noisy supervision signal. Finally, the
performance gap between SAF-IS CCF and SAF-IS CCM is significantly smaller for
the<EMAIL_ADDRESS>metric (-$\Delta$4.48% on average across the two datasets) compared
to the<EMAIL_ADDRESS>metric (-$\Delta$9.74%). This can be attributed to the lower
quality of FUN-SIS binary segmentation masks, causing a performance drop when
a high IoU threshold is used: the lower 50% IoU threshold, instead, being less
affected by possible inaccuracies in the binary segmentation masks, highlights
the high instantiation quality.

###  5.2 Tool Instance Segmentation

In order to evaluate instance segmentation results, and compare them with
other state-of-the-art segmentation approaches, we adopt the commonly used IoU
EndoVis challenge metric defined in [12]. It is worth noticing that such
metric treats the segmentation problem as pixel-wise classification, without
providing information about instantiation quality. Table 2 reports results of
our SAF-IS framework and for several state-of-the-art solutions. For each
method the table highlights the type of supervision used for training. State-
of-the-art approaches are all trained in a fully-supervised manner using
pixel-level semantic annotations (S), in combination with pixel-level instance
annotations for instance segmentation methods (I). Our SAF-IS framework does
not require pixel-level semantic or instance annotations to train, relying
instead only on prototype instance labels (P) - 8 for the experiments reported
in this Table - and weak labels, in the form of frame-wise (FW) or sequence-
wise (SW) tool presence labels (results for both modalities are reported). In
addition SAF-IS can be trained using manually annotated binary masks (B) if
available, or rely on the predictions of the unsupervised FUN-SIS approach
(results for both modalities are also reported).
Results presented in Table 2 show that our SAF-IS approach, trained using only
binary tool presence labels and 8 prototype labels, outperforms fully-
supervised and semi-supervised solutions adopting a semantic segmentation
problem formulation (Ternaus, MF-TN, DMF-TN), despite not requiring any
spatial annotation. On the EndoVis 2017 dataset our solution also outperforms
a standard Mask-RCNN (MRCNN), trained on manually annotated segmentation masks
and bounding-boxes for ground truth instantiation. In addition to pixel-level
semantic and instance annotations, the solutions outperforming our SAF-IS
approach also rely on temporal information during inference (†) and additional
tool-part segmentation annotations (‡). It is worth noticing that temporal
modelling is a natural extension for SAF-IS, as tool tracking information is
already extracted as part of the instance-wise feature learning step. Finally,
a comparison between SAF-IS models trained on frame-wise (FW) and sequence-
wise (SW) binary tool presence labels, shows the effectiveness of our teacher-
student solution to extract a reliable supervision signal from the
automatically obtainable sequence-wise labels, with an average gap between the
two of less than 1.2% IoU, across datasets and binary mask sources.
Qualitative results are shown in Figures 12 & 13 at the end of the manuscript.

Fig. 8: Impact of grid square resolution and threshold value $\varepsilon_{C}$
on the tool instantiation quality for the EndoVis2017 dataset (left) and
EndoVis2018 dataset (right). The combination used in our main experiments is
highlighted in red.

Fig. 9: From left to right, original image, predicted displacement field, and
examples of centroid regions and instantiation masks for different
combinations of grid resolution and threshold $\varepsilon_{C}$. Mask colors
indicate the ID assigned to the tube each instance belongs to. The combination
adopted in our main experiments is highlighted in red.

##  6 Ablation Studies

In order to provide a deeper insight into the SAF-IS framework, we now present
and discuss ablation studies on three critical design choices: the
augmentation strategy for tool instantiation, the inference parameters for
tool instantiation and the number of prototype labels required for instance
classification.

###  6.1 Tool Instantiation Augmentation Strategy

In order to train the displacement network for instrument instantiation, a
pseudo-supervision signal is generated from the binary masks using a Connected
Component algorithm. Such signal is subsequently refined by 1) preventing
training on potentially overlapping instances (OV) and 2) pasting random tool
instances (PS) to artificially simulate the case of overlapping instances
(Section 3.1).
Table 3 provides results of an ablation study exploring different combinations
of the two augmentation strategies. Such results show the effectiveness of the
two augmentation strategies, and of their simultaneous use. In the case of
binary annotated masks, instance masking (OV) provides an average improvement
of +$\Delta$4.46%<EMAIL_ADDRESS>and +$\Delta$1.08%<EMAIL_ADDRESS>across the two datasets,
compared to the setting where no augmentation is used; instance pasting (PS)
provide an average improvement of +$\Delta$3.03%<EMAIL_ADDRESS>and +$\Delta$1.56%
<EMAIL_ADDRESS>the two strategies combined provide an average improvement of
+$\Delta$7.02%<EMAIL_ADDRESS>and +$\Delta$5.55%<EMAIL_ADDRESS>On the EndoVis 2018 dataset,
paste augmentation appears less effective: this could be due to the fact that
several frames in it present at least 4 separate tool instances, making the
additional pasting redundant, and potentially detrimental as frames can become
too cluttered.

Fig. 10: Left: visualization of the learnt feature representations of the
EndoVis 2017 training set instances, projected in the 2D space using t-SNE
algorithm [27]. Each instance point is colored according to the corresponding
ground truth tool class. Right: K-Means++ clustering and prototype labels
obtained using different number of clusters $N_{km}$; projected features and
prototype instances are colored accordingly to the corresponding prototype
labels.

$N_{km}$ | K-Means | Teacher | Student
---|---|---|---
SW | FW
8 | 43.86 | 45.66 | 52.64 | 53.73
16 | 38.52 | 41.34 | 50.02 | 52.64
32 | 42.33 | 45.26 | 51.23 | 52.44
64 | 44.76 | 48.38 | 52.84 | 53.37
(a)

$N_{km}$ | K-Means | Teacher | Student
---|---|---|---
SW | FW
8 | 30.47 | 30.88 | 42.21 | 45.86
16 | 37.86 | 41.81 | 46.95 | 47.33
32 | 32.49 | 36.40 | 46.40 | 48.00
64 | 36.10 | 41.02 | 46.91 | 47.96
(b)

$N_{km}$ | K-Means | Teacher | Student
---|---|---|---
SW | FW
8 |  56.62 | 56.80 | 63.57 | 63.38
16 | 56.03 | 57.25 | 60.63 | 61.96
32 | 56.11 | 57.48 | 62.80 | 64.76
64 | 53.80 | 57.22 | 62.24 | 63.88
(c)

$N_{km}$ | K-Means | Teacher | Student
---|---|---|---
SW | FW
8 | 54.08 | 54.14 | 57.75 | 58.03
16 | 56.53 | 57.04 | 57.40 | 58.53
32 | 55.53 | 55.86 | 58.45 | 59.48
64 | 55.52 | 56.02 | 57.92 | 59.85
(d)

Table 4: Results of the ablation study investigating the impact of the number
of clusters $N_{km}$ on final segmentation results, using, for instance
classification, direct K-Means inference, Teacher predictions and Student
predictions, trained using sequence-wise (SW) or frame-wise (FW) binary tool
presence labels. Results obtained using a): manually annotated binary masks on
the EndoVis2017 dataset, b): FUN-SIS predicted binary masks on the EndoVis2017
dataset, c): manually annotated binary masks on the EndoVis2018 dataset, d):
FUN-SIS predicted binary masks on the EndoVis2018 dataset. Segmentation
results were evaluated using the challenge IoU metric. The best results across
the number of clusters are highlighted in bold.

###  6.2 Tool Instantiation Inference Parameters

In order to obtain instance masks, a square grid is overlapped to the
predicted displacement field; centroid squares are then selected as the ones
whose per-pixel average of vectors pointing inside them is greater than the
threshold value $\varepsilon_{C}$. The grid resolution (equal to $32\times 32$
in our main experiments) and the threshold $\varepsilon_{C}$ (equal to 5 in
our main experiments) regulate the trade-off between precision and recall of
the obtained instance masks. We experimentally evaluate the impact of the two
parameters by varying them in a grid-like manner, with grid resolution in
$[8,16,32,64,128]$ and $\varepsilon_{C}$ in $[1,3,5,7,10]$. Their different
combinations are used to obtain instance masks from the same displacement
fields. The<EMAIL_ADDRESS>between the obtained masks and the ground truth instances
is reported in Figure 8 for both the EndoVis2017 and EndoVis2018 datasets.
The presented results, together with the qualitative results shown in Figure
9, clearly highlight the impact of the two parameters. For intermediate grid
resolution values ($32\times 32$, $64\times 64$), the impact of
$\varepsilon_{C}$ is minimal. However, as the grid solution decreases
($16\times 16$, $8\times 8$), an high value of $\varepsilon_{C}$ negatively
affects the quality as instantiation, as the average convergence rate on large
squares tends to be lower. This can be also observed from the qualitative
instantiation results shown in Figure 9, top-right, where no candidate squares
reach the threshold. Vice-versa, high grid resolution values ($128\times 128$)
tend to be more negatively affected by a low $\varepsilon_{C}$, as it leads to
the identification of many false positive centroids (instantiation results
from Figure 9, bottom-left).

###  6.3 Prototype Labels Number

In SAF-IS, the Teacher network is required to gather knowledge from the
prototype labels, in order to be able to identify the correct ordered sets of
weak labels used for Student training. Prototype labels, therefore can have a
crucial influence on the quality of instance classification. In addition, they
represent the manual annotation necessarily required by SAF-IS for training,
as both binary tool masks and binary tool presence labels can be automatically
obtained. Therefore we now present, in Table 4, the impact on the segmentation
performance, of the number of clusters $N_{km}$ used for K-Means clustering,
equal to the number of prototype labels assigned by a human operator. In order
to provide a complete overview, we present segmentation results obtained via
instance classification by direct K-Means inference, Teacher classifier
prediction and Student classifier prediction, when trained using sequence-wise
or frame-wise binary tool presence labels. In addition, Figure 10 provides a
visualization of the learnt feature distribution, the clustering process and
the automatically selected prototype instances.
Result analysis provides different insights into the method. First of all,
although a marginal improvement exists, increasing the number of prototype
instances does not provide substantial performance gains for the Student
network. This result may indicate that effective feature learning is a crucial
methodological bottleneck, which cannot be solved by simply increasing the
number of human-assigned labels. Secondly, the presented results highlight the
consistent improvement in performance provided by the Student network, trained
on the weak labels matched through the Teacher model. Although the Teacher
learns to substantially replicate K-Means clustering classification, as shown
by their similar performance, this is enough to perform a good weak label
matching, responsible for Student’s superior performance.

##  7 Discussion

The results presented in Sections 5 & 6 confirm the soundness of the proposed
SAF-IS framework for instance segmentation. Our solution trains on endoscopic
videos paired with binary segmentation masks, potentially obtained in an
unsupervised way, and is designed to incorporate binary tool presence labels,
either frame-wise or sequence-wise. Human annotation effort can here be
limited to labelling a tiny set of prototype instances, automatically selected
by our approach, with inexpensive classification labels: the ablation study
presented in Section 6 shows that the size of such set can be reduced to 8
instances ($\sim$0.26% of the total number of training instances), with no
significant performance drop. This result goes significantly beyond existing
semi-supervised solutions like [51], where a significant set of frames (up to
30%) needs to be labelled with pixel-level annotations, while still providing
inferior segmentation performance. Indeed, our complete spatial annotation-
free solution, using FUN-SIS predicted binary masks for training, outperforms
fully-supervised and semi-supervised semantic segmentation approaches like MF-
TN and DMF-TN by a consistent margin on the EndoVis 2017 dataset. Furthermore,
our SAF-IS framework effectively incorporates sequence-wise binary tool
presence labels, commonly overlooked in the literature. This small gap in
performance between frame-wise and sequence-wise training modalities (Table
2), shows that sequence-wise labels can be an effective source of supervision,
while being completely free to collect.
Although a performance gap still exists with top-performing fully-supervised
instance segmentation approaches, we believe there exist several directions of
improvement to close such a gap. First of all, temporal modelling could be
easily learnt from the already available tracking information, currently
exploited only at training time for feature learning. Secondly, as highlighted
by the ablation study on cluster number, feature learning represents a crucial
methodological bottleneck: if the learnt feature representations are sub-
optimal, the unsupervised clustering may fail to separate tools belonging to
different classes, hindering the following classifier training. In the current
implementation, feature learning is performed in a completely unsupervised
way, with no help from external information. Weak information about binary
tool presence may be included at this stage to perform a more informed
positive and negative feature sampling.

Fig. 11: Top row: SAM [20] segmentation results on the EndoVis2017 dataset.
Central row: SAF-IS instantiation results obtained from binary manually
annotated masks. Bottom row: SAF-IS instantiation results obtained from FUN-
SIS predicted binary masks. Each instance is colored using a random color,
which is not meant to represent tool type classes.

In addition to these direct improvements, SAF-IS, not requiring pixel-level
labels, can leverage recent break-through solutions like SAM (Segment Anything
Model, [20]) to directly obtain instance-wise masks for the following feature
learning and tool classification training. Figure 11 shows qualitative results
from SAM (without text prompts, not yet released at the time of this
submission) on the EndoVis2017 dataset, compared to SAF-IS predictions. Even
if SAM segmentation results are currently over-segmenting tools, breaking them
up into individual parts, our SAF-IS instantiation predictions could be used
to group these parts, exploiting the high-quality boundary segmentation that
SAM can already provide.
In conclusion, SAF-IS major contribution lies in its ability to lift the need
for spatial annotation of the training data. This may open up new research
directions aimed at better exploiting human annotation effort, for example by
focusing it on particularly representative or challenging samples.

Fig. 12: Qualitative segmentation results from the EndoVis2017 dataset,
highlighting, for our SAF-IS approach, the source of binary segmentation masks
(BS) and the type of binary tool presence labels (frame-wise or sequence-
wise). All the SAF-IS results are obtained using 8 prototype labels. Row 1:
ground truth; rows 2-5: SAF-IS Student trained on (2) manually annotated
binary masks and frame-wise tool presence labels, (3) manually annotated
binary masks and sequence-wise tool presence labels, (4) FUN-SIS predicted
binary masks and frame-wise tool presence labels, (3) FUN-SIS predicted binary
masks and sequence-wise tool presence labels.

Fig. 13: Qualitative segmentation results from the EndoVis2018 dataset. Ground
truth and SAF-IS Student results presented in the same order as Figure 12
above.

##  8 Conclusion

In this work we developed and validated SAF-IS, a Spatial Annotation Free
framework for Instance Segmentation of surgical instruments. The proposed
framework embraces recent breakthrough solutions for unsupervised binary
segmentation, building on top of them to perform instance segmentation without
requiring pixel-level semantic or instance annotations to train. Instead, SAF-
IS exploits the binary tool masks to learn to encode each instance in a
compact feature representation, and solves the instance classification problem
by relying on cheaply obtainable binary tool presence labels. A supplementary
video highlighting crucial methodological aspects and providing additional
qualitative results is available at https://vimeo.com/860204311. In
conclusion, we hope this work can show the potential of prior knowledge and
weakly-supervised training for tool instance segmentation, encouraging the
search for alternatives to full-supervision for increasingly complex surgical
computer vision tasks.

## Acknowledgments

This work was supported by the ATLAS project. The ATLAS project has received
funding from the European Union’s Horizon 2020 research and innovation
programme under the Marie Sklodowska-Curie grant agreement No. 813782. This
work was also partially supported by French State Funds managed by the Agence
Nationale de la Recherche (ANR) through the Investissements d’Avenir Program
under Grant ANR-11-LABX-0004 (Labex CAMI) and Grant ANR-10-IAHU-02 (IHU-
Strasbourg), by French state funds managed by the ANR under references
ANR-20-CHIA-0029-01 (National AI Chair AI4ORSafety) and ANR-18-CE19-0012, and
by BPI France under reference DOS0180017/00 (project 5G-OR).

## References

  * Allan et al. [2020] Allan, M., Kondo, S., Bodenstedt, S., Leger, S., Kadkhodamohammadi, R., Luengo, I., Fuentes, F., Flouty, E., Mohammed, A., Pedersen, M., et al., 2020\.  2018 robotic scene segmentation challenge.  arXiv preprint arXiv:2001.11190 .
  * Allan et al. [2018] Allan, M., Ourselin, S., Hawkes, D.J., Kelly, J.D., Stoyanov, D., 2018.  3-d pose estimation of articulated instruments in robotic minimally invasive surgery.  IEEE transactions on medical imaging 37, 1204–1213.
  * Allan et al. [2019] Allan, M., Shvets, A., Kurmann, T., Zhang, Z., Duggal, R., Su, Y.H., Rieke, N., Laina, I., Kalavakonda, N., Bodenstedt, S., et al., 2019\.  2017 robotic instrument segmentation challenge.  arXiv preprint arXiv:1902.06426 .
  * Arthur and Vassilvitskii [2006] Arthur, D., Vassilvitskii, S., 2006\.  k-means++: The advantages of careful seeding.  Technical Report. Stanford.
  * Bolelli et al. [2019] Bolelli, F., Allegretti, S., Baraldi, L., Grana, C., 2019\.  Spaghetti labeling: Directed acyclic graphs for block-based connected components labeling.  IEEE Transactions on Image Processing 29, 1999–2012.
  * Bouget et al. [2015] Bouget, D., Benenson, R., Omran, M., Riffaud, L., Schiele, B., Jannin, P., 2015\.  Detecting surgical tools by modelling local appearance and global shape.  IEEE transactions on medical imaging 34, 2603–2617.
  * Carion et al. [2020] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S., 2020\.  End-to-end object detection with transformers, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, Springer. pp. 213–229.
  * da Costa Rocha et al. [2019] da Costa Rocha, C., Padoy, N., Rosa, B., 2019.  Self-supervised surgical tool segmentation using kinematic information, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE. pp. 8720–8726.
  * Durand et al. [2017] Durand, T., Mordan, T., Thome, N., Cord, M., 2017\.  Wildcat: Weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 642–651.
  * Francis et al. [2018] Francis, N., Curtis, N., Conti, J., Foster, J., Bonjer, H., Hanna, G., 2018\.  Eaes classification of intraoperative adverse events in laparoscopic surgery.  Surgical Endoscopy 32, 3822–3829.
  * Garcia-Peraza-Herrera et al. [2017] Garcia-Peraza-Herrera, L.C., Li, W., Fidon, L., Gruijthuijsen, C., Devreker, A., Attilakos, G., Deprest, J., Vander Poorten, E., Stoyanov, D., Vercauteren, T., et al., 2017\.  Toolnet: holistically-nested real-time segmentation of robotic surgical tools, in: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE. pp. 5717–5722.
  * González et al. [2020] González, C., Bravo-Sánchez, L., Arbelaez, P., 2020.  Isinet: an instance-based approach for surgical instrument segmentation, in: Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23, Springer. pp. 595–605.
  * Hasan and Linte [2019] Hasan, S.K., Linte, C.A., 2019\.  U-netplus: A modified encoder-decoder u-net architecture for semantic and instance segmentation of surgical instruments from laparoscopic images, in: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE. pp. 7205–7211.
  * He et al. [2017] He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017\.  Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, pp. 2961–2969.
  * Islam et al. [2021] Islam, M., Vibashan, V., Lim, C.M., Ren, H., 2021.  St-mtl: Spatio-temporal multitask learning model to predict scanpath while tracking instruments in robotic surgery.  Medical Image Analysis 67, 101837\.
  * Jaiswal et al. [2020] Jaiswal, A., Babu, A.R., Zadeh, M.Z., Banerjee, D., Makedon, F., 2020.  A survey on contrastive self-supervised learning.  Technologies 9, 2\.
  * Jin et al. [2019] Jin, Y., Cheng, K., Dou, Q., Heng, P.A., 2019.  Incorporating temporal prior from motion flow for instrument segmentation in minimally invasive surgery video, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 440–448.
  * Kalia et al. [2021] Kalia, M., Aleef, T.A., Navab, N., Black, P., Salcudean, S.E., 2021.  Co-generation and segmentation for generalized surgical instrument segmentation on unlabelled data, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 403–412.
  * Khosla et al. [2020] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D., 2020.  Supervised contrastive learning.  Advances in neural information processing systems 33, 18661–18673.
  * Kirillov et al. [2023] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al., 2023\.  Segment anything.  arXiv preprint arXiv:2304.02643 .
  * Kong et al. [2021] Kong, X., Jin, Y., Dou, Q., Wang, Z., Wang, Z., Lu, B., Dong, E., Liu, Y.H., Sun, D., 2021.  Accurate instance segmentation of surgical instruments in robotic surgery: Model refinement and cross-dataset evaluation.  International Journal of Computer Assisted Radiology and Surgery 16, 1607–1614.
  * Kurmann et al. [2021] Kurmann, T., Márquez-Neila, P., Allan, M., Wolf, S., Sznitman, R., 2021.  Mask then classify: multi-instance segmentation for surgical instruments.  International journal of computer assisted radiology and surgery 16, 1227–1236.
  * Laina et al. [2017] Laina, I., Rieke, N., Rupprecht, C., Vizcaíno, J.P., Eslami, A., Tombari, F., Navab, N., 2017.  Concurrent segmentation and localization for tracking of surgical instruments, in: International conference on medical image computing and computer-assisted intervention, Springer. pp. 664–672.
  * Lavanchy et al. [2021] Lavanchy, J.L., Zindel, J., Kirtac, K., Twick, I., Hosgor, E., Candinas, D., Beldi, G., 2021.  Automation of surgical skill assessment using a three-stage machine learning algorithm.  Scientific reports 11, 1–9.
  * Lee et al. [2019] Lee, E.J., Plishker, W., Liu, X., Bhattacharyya, S.S., Shekhar, R., 2019.  Weakly supervised segmentation for real-time surgical tool tracking.  Healthcare technology letters 6, 231–236.
  * Liu et al. [2020] Liu, D., Wei, Y., Jiang, T., Wang, Y., Miao, R., Shan, F., Li, Z., 2020.  Unsupervised surgical instrument segmentation via anchor generation and semantic diffusion, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 657–667.
  * Van der Maaten and Hinton [2008] Van der Maaten, L., Hinton, G., 2008\.  Visualizing data using t-sne.  Journal of machine learning research 9\.
  * Mascagni et al. [2022] Mascagni, P., Vardazaryan, A., Alapatt, D., Urade, T., Emre, T., Fiorillo, C., Pessaux, P., Mutter, D., Marescaux, J., Costamagna, G., et al., 2022\.  Artificial intelligence for surgical safety: automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning.  Annals of surgery 275, 955–961.
  * Meinhardt et al. [2022] Meinhardt, T., Kirillov, A., Leal-Taixe, L., Feichtenhofer, C., 2022\.  Trackformer: Multi-object tracking with transformers, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8844–8854.
  * Nwoye et al. [2019] Nwoye, C.I., Mutter, D., Marescaux, J., Padoy, N., 2019\.  Weakly supervised convolutional lstm approach for tool tracking in laparoscopic videos.  International journal of computer assisted radiology and surgery 14, 1059–1067.
  * Padoy et al. [2012] Padoy, N., Blum, T., Ahmadi, S.A., Feussner, H., Berger, M.O., Navab, N., 2012\.  Statistical modeling and recognition of surgical workflow.  Medical image analysis 16, 632–641.
  * Pakhomov et al. [2019] Pakhomov, D., Premachandran, V., Allan, M., Azizian, M., Navab, N., 2019.  Deep residual learning for instrument segmentation in robotic surgery, in: International Workshop on Machine Learning in Medical Imaging, Springer. pp. 566–573.
  * Pakhomov et al. [2020] Pakhomov, D., Shen, W., Navab, N., 2020.  Towards unsupervised learning for instrument segmentation in robotic surgery with cycle-consistent adversarial networks, in: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE. pp. 8499–8504.
  * Ramesh et al. [2022] Ramesh, S., Srivastav, V., Alapatt, D., Yu, T., Murali, A., Sestini, L., Nwoye, C.I., Hamoud, I., Fleurentin, A., Exarchakis, G., et al., 2022\.  Dissecting self-supervised learning methods for surgical computer vision.  arXiv preprint arXiv:2207.00449 .
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., Brox, T., 2015.  U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, Springer. pp. 234–241.
  * Ross et al. [2018] Ross, T., Zimmerer, D., Vemuri, A., Isensee, F., Wiesenfarth, M., Bodenstedt, S., Both, F., Kessler, P., Wagner, M., Müller, B., et al., 2018\.  Exploiting the potential of unlabeled endoscopic video data with self-supervised learning.  International journal of computer assisted radiology and surgery 13, 925–933.
  * Russo [2012] Russo, M., 2012.  Triangulation concept for minimally invasive access surgery.  US Patent App. 13/442,006.
  * Sahu et al. [2020] Sahu, M., Strömsdörfer, R., Mukhopadhyay, A., Zachow, S., 2020\.  Endo-sim2real: Consistency learning-based domain adaptation for instrument segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 784–794.
  * Sestini et al. [2021] Sestini, L., Rosa, B., De Momi, E., Ferrigno, G., Padoy, N., 2021.  A kinematic bottleneck approach for pose regression of flexible surgical instruments directly from images.  IEEE Robotics and Automation Letters 6, 2938–2945.
  * Sestini et al. [2023] Sestini, L., Rosa, B., De Momi, E., Ferrigno, G., Padoy, N., 2023.  Fun-sis: A fully unsupervised approach for surgical instrument segmentation.  Medical Image Analysis , 102751.
  * Shvets et al. [2018] Shvets, A.A., Rakhlin, A., Kalinin, A.A., Iglovikov, V.I., 2018\.  Automatic instrument segmentation in robot-assisted surgery using deep learning, in: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE. pp. 624–628.
  * Tanzi et al. [2021] Tanzi, L., Piazzolla, P., Porpiglia, F., Vezzetti, E., 2021\.  Real-time deep learning semantic segmentation during intra-operative surgery for 3d augmented reality assistance.  International Journal of Computer Assisted Radiology and Surgery 16, 1435–1445.
  * Twinanda et al. [2016] Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N., 2016\.  Endonet: a deep architecture for recognition tasks on laparoscopic videos.  IEEE transactions on medical imaging 36, 86–97.
  * Vardazaryan et al. [2018] Vardazaryan, A., Mutter, D., Marescaux, J., Padoy, N., 2018\.  Weakly-supervised learning for tool localization in laparoscopic videos, in: Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3, Springer. pp. 169–179.
  * Voros et al. [2006] Voros, S., Long, J.A., Cinquin, P., 2006.  Automatic localization of laparoscopic instruments for the visual servoing of an endoscopic camera holder, in: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2006: 9th International Conference, Copenhagen, Denmark, October 1-6, 2006. Proceedings, Part I 9, Springer. pp. 535–542.
  * Wang et al. [2022] Wang, Y., Long, Y., Fan, S.H., Dou, Q., 2022.  Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 431–441.
  * Wei et al. [1997] Wei, G.Q., Arbter, K., Hirzinger, G., 1997.  Automatic tracking of laparoscopic instruments by color coding, in: CVRMed-MRCAS’97: First Joint Conference Computer Vision, Virtual Reality and Robotics in Medicine and Medical Robotics and Computer-Assisted Surgery Grenoble, France, March 19–22, 1997 Proceedings, Springer. pp. 357–366.
  * Xie et al. [2021] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P., 2021.  Segformer: Simple and efficient design for semantic segmentation with transformers.  Advances in Neural Information Processing Systems 34, 12077–12090.
  * Xue et al. [2022] Xue, Y., Liu, S., Li, Y., Wang, P., Qian, X., 2022.  A new weakly supervised strategy for surgical tool detection.  Knowledge-Based Systems 239, 107860\.
  * Yang et al. [2022] Yang, Z., Simon, R., Linte, C., 2022.  A weakly supervised learning approach for surgical instrument segmentation from laparoscopic video sequences, in: Medical Imaging 2022: image-Guided Procedures, Robotic Interventions, and Modeling, SPIE. pp. 412–417.
  * Zhao et al. [2020] Zhao, Z., Jin, Y., Gao, X., Dou, Q., Heng, P.A., 2020\.  Learning motion flows for semi-supervised instrument segmentation from robotic surgical video, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 679–689.
  * Zhao et al. [2022] Zhao, Z., Jin, Y., Heng, P.A., 2022.  Trasetr: track-to-segment transformer with contrastive query for instance-level instrument segmentation in robotic surgery, in: 2022 International Conference on Robotics and Automation (ICRA), IEEE. pp. 11186–11193.