# TraceNet: Segment one thing efficiently

Mingyuan Wu1, Zichuan Liu2, Haozhen Zheng1, Hongpeng Guo1, Bo Chen1, Xin Lu2,
Klara Nahrstedt1 1University of Illinois at Urbana-Champaign, Urbana, IL, USA
2Independent Researcher, USA

###### Abstract.

Efficient single instance segmentation is essential for unlocking features in
the mobile imaging applications, such as capture or editing. Existing on-the-
fly mobile imaging applications scope the segmentation task to portraits or
the salient subject due to the computational constraints. Instance
segmentation, despite its recent developments towards efficient networks, is
still heavy due to the cost of computation on the entire image to identify all
instances. To address this, we propose and formulate a one tap driven single
instance segmentation task that segments a single instance selected by a user
via a positive tap. This task, in contrast to the broader task of segmenting
anything as suggested in the Segment Anything Model (Kirillov et al., 2023),
focuses on efficient segmentation of a single instance specified by the user.
To solve this problem, we present TraceNet, which explicitly locates the
selected instance by way of receptive field tracing. TraceNet identifies image
regions that are related to the user tap and heavy computations are only
performed on selected regions of the image. Therefore overall computation cost
and memory consumption are reduced during inference. We evaluate the
performance of TraceNet on instance IoU average over taps and the proportion
of the region that a user tap can fall into for a high-quality single-instance
mask. Experimental results on MS-COCO and LVIS demonstrate the effectiveness
and efficiency of the proposed approach. TraceNet can jointly achieve the
efficiency and interactivity, filling in the gap between needs for efficient
mobile inference and recent research trend towards multimodal and interactive
segmentation models.

††copyright: none

##  1\. Introduction

The development of image segmentation has never been stopped in the past
decades due to its critical role in image understanding applications (LLC,
2023; Amazon, 2023; ps_camera; Microsoft, 2020). The initial goal of image
segmentation is to segment out objects of certain semantic categories as
defined in the tasks of semantic segmentation and instance segmentation. Most
literature under the umbrella of these tasks (Hu et al., 2021; He et al.,
2017; Cheng et al., 2022) have been focusing on improving the segmentation
accuracy. To further aid segmentation accuracy, user inputs have been
considered as additional cues and modalities. In particular, traditional
click-based interactive image segmentation optimizes for the minimal number of
clicks to achieve a satisfactory segmentation result on user specified
objects. Going beyond clicks, latest large promptable segmentation models
(Kirillov et al., 2023; Zou et al., 2023) are designed and trained to take a
variety of user queries including boxes, scribbles, masks and free-form texts.
These models demonstrate strong zero-shot generalization to new downstream
vision tasks and new image distributions. At the same time, these models
enable more segmentation pipelines to involve multimodal user inputs, and
significantly increase interactivity in the inference stage.

Despite a trend towards multimodal and interactive segmentation, mobile
applications, as natural interfaces to obtain user prompts, have not fully
enjoyed recent advancement of the segmentation algorithms. A gap exists
between the inefficiency and heavy computation of the segmentation algorithms
and limited computational resources in mobile scenarios. To bridge the gap,
researchers have been investigating efficient approaches for segmentation so
that the algorithms can run smoothly in mobile editing apps or even capture
apps. Efficient segmentation approaches (Cao et al., 2020; Lee and Park, 2020;
Wang et al., 2020; Xie et al., 2019; Tian et al., 2020; Bolya et al., 2020a,
b) usually are build upon efficient neural network architectures (Sandler et
al., 2018; Howard et al., 2019; Chen et al., 2018). In practical cases that
require extreme efficiency, mobile applications with these algorithms limit
the number of semantic categories, or focus on the salient subject in the
scene. A representative successful application of efficient segmentation
approaches is the portrait mode that is widely supported in default camera
apps on mobile phones111iphone portrait mode: https://support.apple.com/en-
us/HT208118, Pixel6 portrait mode:
https://store.google.com/us/magazine/pixel_camera?hl=en-US. This mode employs
portrait segmentation techniques, generally limited to binary semantic
categories such as person versus background or salient object versus
background, to enhance efficiency. As far as we know, the potential for a
facilitative relationship between interactivity and efficiency and jointly
achieving both of them remains unexplored in segmentation literature and
industry applications. In this context of efficient and interactive
segmentation for mobile imaging applications, we propose to use user inputs
(such as an arbitrary tap on the instance) to enable efficient single-instance
segmentation. The proposed formulation relaxes the limitation of existing
efficient segmentation that applies only to the salient subject in the scene.
As shown in Figure 1(a), with a tap of the dog, unicorn, or pumpkin, we expect
to get its segmentation mask instantly. With the tap-based single-instance
segmentation, automatic images manipulation, such as depth-of-field effects,
background replacement and image enhancement, therefore can be enabled on an
arbitrary instance in the image.

Figure 1. Tap-driven Single-Instance Segmentation: (a) A single instance mask
is produced when a user taps an instance in the viewfinder. (b) The clickable
region supported by our algorithm is highlighted in blue and the predicted
instance mask is highlighted in yellow. Various local adjustments are expected
to happen in the yellow area before the image is captured in the mobile
applications.

That is, we strive for a more specific type of segmentation model for mobile
applications and answer a question: since users have already indicated
interactively what object they like on their phones, what should our efficient
segmentation model do smartly. Formally, We propose a novel form of image
segmentation task, namely one tap efficient segmentation, which is motivated
by the demands of on-the-fly image editing mobile applications. The task aims
at segmenting one instance that a user queries with a positive tap in an
extremely efficient manner. We use ”taps” to better represent intuitive user
interactions with modern mobile applications, as opposed to ”clicks”
associated with mouse-driven desktop environments. It is worth mentioning that
modern mobile devices register a tap as a specific pixel input, rather than a
broader area in the image. In practical scenarios of one-tap efficient
segmentation, a user’s tap might not always accurately target the center of
the desired instance. The imperfect click might result in a unsatisfying
segmentation result that affects user experience. To improve user experience,
we propose a metric of user tap tolerance in addition to segmentation
accuracy. This metric measures the proportion of the region that a user tap
can fall into for a high-quality single-instance mask, as shown in Figure 1
(b).

Our research insight towards the proposed problem is that the user tap could
serve as a cue to guide the model to perform inference in a selective manner.
Following the insight, we propose a straightforward yet effective solution
TraceNet, which employs receptive field tracing through the neural network to
explicitly locate the selected instance, thereby avoiding the need for
intensive feature computation across the entire image. TraceNet is inspired by
recently proposed conditional instance segmentation methods (Tian et al.,
2020; Yu et al., 2022), which predicts local query features around the single
instance and dynamically conditions the instance-aware mask head on the local
features. Based on the conditional mechanism, TraceNet can dynamically control
inference given a tap query. Since local relevant features around the user tap
of the instance are important to guide the segmentation prediction, a
receptive-field tracer (RFT) is introduced to control the spatial region where
heavy computation is applied to. The controller back-traces computation
dependency across each layer and crops out all pixels that do not contribute
to the local features around the tap. The cropping is conducted in the
forwarding pass, eliminating spatially redundant convolution operations.
Overall our contribution can be summarized as follows:

  * •

Motivated by real-world demands, we proposed and formulated a one tap
efficient segmentation as a new form of efficient segmentation for a single
instance, and designed evaluation protocols.

  * •

We propose a solution of TraceNet for one tap efficient segmentation. TraceNet
conditions the instance-aware mask head on local features around the user’s
click and efficiently controls the usage of local features.

  * •

We evaluated the proposed TraceNet on MS-COCO (Lin et al., 2014) and LVIS
(Gupta et al., 2019). TraceNet demonstrates high computational efficiency
while achieving high accuracy on the mask prediction of user-specified
instances along with a high user tap tolerance.

##  2\. Related Work

The proposed problem of one tap efficient segmentation aligns closely with a
broad line of segmentation research that leverages user clicks. We review
studies in these categories and discuss differences of our target. Meanwhile,
as one tap segmentation can be achieved by performing segmentation of all
instances, we review studies that directly predict dense instance masks and
discuss the efficiency advantage of the proposed method. Lastly, the proposed
TraceNet efficiently makes use of relevant image features with regard to the
user tap, which is related to studies that aim to avoid applying heavy
convolutions on the entire images.

###  2.1. Click-based Interactive Segmentation

Most of existing interactive segmentation pipelines in the literature optimize
for the minimal user clicks so that the IoU (Intersection over Union) between
the predicted foreground mask and the groundtruth mask exceeds a pre-defined
threshold. Classic methods in this field typically utilize low-level image
features and the properties of clicks. Well-known traditional approaches
include Intelligent Scissors (Mortensen and Barrett, 1995, 1998), Graphcut
(Boykov and Jolly, 2001), Grabcut (Rother et al., 2004), and Random-walk
(Grady, 2006). The advent of CNN-based interactive segmentation was marked by
the introduction of DIOS (Xu et al., 2016). This method translates user clicks
into Euclidean distance maps, which are then merged with the original RGB
channels for input into a fully convolutional network (Long et al., 2015). In
recent years, the paradigm of clicks encoding and concatenation are widely
followed by further works in various learning-based interactive segmentation
schemes(Jang and Kim, 2019; Sofiiuk et al., 2020; Liu et al., 2022b; Lin et
al., 2020; Chen et al., 2022; Sofiiuk et al., 2022; Zhou et al., 2023; Liu et
al., 2023). In addition to these specialized interactive segmentation models,
vision foundation models have also demonstrated capability in this area. SAM
(Kirillov et al., 2023), in particular, has attracted significant attention
for its remarkable zero-shot generalization to new image distributions and
tasks. SAM operates by conditioning its mask decoder on combined output
embeddings from a heavy image encoder and a prompt encoder, inherently
functioning as an interactive segmenter for any instance. MobileSAM (Zhang et
al., 2023) makes SAM more mobile-friendly by distilling the knowledge from
SAM’s heavy encoder into a more lightweight one. However, our task differs
from conventional interactive segmentation, as it focuses on efficiently
segmenting a single instance rather than segmenting anything or everything.
Guided by a user’s tap within one instance, TraceNet can selectively encodes
the instance-relevant features and thus benefit from efficiency gains as it
avoids redundant all-instance aware feature calculation across the entire
image.

###  2.2. Image-level Mask Predictions

Recent efficient instance segmentation approaches take the entire image as
input and predict dense instance masks directly (Yu et al., 2022; Tian et al.,
2020; Bolya et al., 2020a). Unlike Mask-RCNN (He et al., 2017), these
approaches avoid ROI operations to obtain local feature maps and are more
efficient as a result. YOLACT+ (Bolya et al., 2020a) breaks the segmentation
task into generating protoype masks and directly predicting per-instance mask
coefficients. Condinst(Tian et al., 2020) adds a dynamic instance-aware mask
branch to the object detector FCOS(Tian et al., 2021). With parameters
conditioned on local features individual instance, the mask branch can be
directly applied to predict instance masks from the whole feature map .
SOIT(Yu et al., 2022) is a follow-up conditional image-level prediction
algorithm that applies encoder-decoder transformer architecture to produce
instance-related query features. While these image-level prediction algorithms
generally achieve a much faster inference speed, there is room for efficiency
improvement, especially only one specific instance mask is queried by users.
TraceNet addresses this gap, offering a principled approach to enhance the
efficiency of these models in one-instance cases.

###  2.3. Spatial Redundancy of Convolution

In most of the convolutional neural networks, convolution is the most
computational heavy operator that impacts inference time. Reducing computation
cost by applying convolution on selected areas of images has been successfully
pursued. For example, in the task of small/multi-level object detection,
AutoFocus(Najibi et al., 2019) targets multi-level efficient inference. It
predicts and crop segmentation map at coarser scales, and then runs the
detector at finer scales on a small fraction of the entire image.
QueryDet(Yang et al., 2021) predicts the coarse locations of small objects on
low-resolution features, and then runs sparse convolution on high-resolution
feature spatially guided by the coarser predictions. In AutoFocus and
QueryDet, “focus cropping” operations are conducted either on images or
feature maps, improving inference speed of (small) object detection. The gain
of inference speed is led by the sparsity of small objects.

##  3\. Task and Method

Problem Formulation. Given an input image $I\in\mathbb{R}^{H\times W\times 3}$
and a user tap $c=(x,y)\in\mathbb{R}^{H\times W}$. The goal of tap-driven
single-instance segmentation is to predict a pixel-level mask of the instance
located at $(x,y)$. The ground-truth is defined by $\left\\{M_{gt}\right\\}$,
where $M_{gt}\in\\{0,1\\}^{H\times W}$ is the groundtruth mask that user
queries with the tap $c$, where $c\in M_{gt}$. The expected model is a learned
function that maps $I$ and $c$ to a single-instance mask $M_{pred}$, targeting
a satisfying IoU between $M_{pred}$ and $M_{gt}$. Because $M_{pred}$ depends
on the user tap $c$, we propose a new metric, the mean tap Intersection over
Union (mIoU-T), to measure the average segmentation accuracy of all possible
user taps within the groundtruth instance mask. The user tap tolerance is
measured by the newly proposed metric mean Tap Area (mTA). mTA calculates the
ratio between area of feasible taping area and the area of groundtruth
instance mask. The feasible taping area covers potential taps that can
generate an instance mask with the IoU over a predefined threshold. We refer
readers to the experiment section 4.2 for the formula of mIoU-T and mTA.

Overall System Architecture. As shown in Figure 2, TraceNet introduces a key
component named Receptive Field Tracer (RFT). The Receptive Field Tracer
targets eliminating the computation redundancy in the exhausted searching for
instances in existing instance segmentation algorithms towards tap-driven
segmentation. Given a user tap, the RFT back-traces the receptive field region
at each layer with respect to local features around the user query tap. The
receptive field back-tracing can be regarded as a Depth-First Search process
that recursively computes receptive fields from downstream layers to upper-
stream layers. Based on the derived receptive fields, the features that do not
contribute to the output related to the user tap will be removed during the
inference phase. Thus, RFT reduces the computation by controlling the
inference flow of the most parts of an instance segmentation model. During the
inference phase, the RFT removes the redundant computation that does not
contribute to the output, which brings significant performance improvement in
running speed and memory consumption. The RFT can be applied to convolutional
instance segmentation algorithms in general.

Figure 2. \- The overall architecture of TraceNet. Based on a User Tap,
Receptive Field Tracer back traces the receptive field region and perform
autocropping in all convolution heads except the mask branch. Details of
receptive field backtracing, autocropping and other modules for instance
segmentation are discussed in section 3.

###  3.1. Receptive Field Back-Tracing in TraceNet

Figure 3. One tap driven receptive field back tracing in the acyclic
computation graph (visualized directly by onnx) of instance segmentation. The
receptive field is backtraced recursively with post-order Depth-first search.
Computations in grey regions are significanty reduced with autocropping
mechanism

As it is shown in Figure 3, receptive field of each layer in the computation
graph of instance segmentation is backtraced recursively, enabling
autocropping function in the forwarding pass and saving computations. Details
will be discussed in this section. The section is organized as follows: we
will first give a formal definition of receptive field region at each layer in
neural networks, then give a detail post-order Depth First Search algorithm
illustrating the mechanism of receptive field back-tracing in arbitrary pure
convolution neural network models, derive all the recursive cases and finally
explain how Receptive Field Tracer makes use of the results to control the
calculation of other components during the inference process.

Receptive Field Revisited. In the context of deep learning(Araujo et al.,
2019; Luo et al., 2016), _receptive field region_ refers to the region in the
input that produces the feature. _Receptive field_ is defined as the size of
the region. The concept of receptive field is important for researchers to
diagnose how CNN works in a sense that a unit in the model output is only
affected by units in receptive field regions in the input image. A formal
definition of the receptive field region of a simple $n$-layer convolutional
neural network can be formulated as follows. Assume the pixels on each layer
are indexed by $(i,j)$, with the most upper-left pixel at $(0,0)$. Denote the
$(i,j)$th pixel on the $p$th layer as $x_{i,j}^{p}$. $p\in[n]$ in a $n$-layer
convolutional neural network where $x_{i,j}^{0}$ and $x_{i,j}^{n}$
respectively denote the pixel value in the input image and the model output.
By definition the receptive field region of the unit $x_{i,j}^{n}$ is the set
of all units in $x^{0}$ that contribute to $x_{i,j}^{n}$. The receptive field
region of a set of units is the union of the receptive field region of all
units in the set. We extend the concept of the receptive field more than in
the input images and define the $p$-layer receptive field region $r^{p}$ of
the unit $x_{i,j}^{n}$ to be the set of all units in the output feature map of
the pth layer $x^{p}$ that contribute to $x_{i,j}^{n}$, for any $p\in[n]$.
Note that we can consider only single channel of the input and output of each
layer in the context of calculating receptive field regions and similar
results can be derived for layers with multiple channels.

Receptive Field Depth First Search. The tap-driven segmentation aims at
precisely localizing the region that contributes to local output features
around user taps and reducing spatial redundancy of convolution operations as
much as possible across all the layers of the model. i.e only computation
within the layer receptive field region is preserved in forward pass. We
introduce a tap-driven receptive field back-tracing algorithm to compute
receptive field regions at each layer of the neural network from deep to
shallow. To illustrate the algorithm, we construct a directed acyclic
computation graph for arbitrary model, where the nodes correspond to layer
operations and inputs, and directed edges represent dependency between layers.
(Note that most modern convolutional neural networks designs (He et al.,
2016)(Lin et al., 2017a) rely on layers with multiple child nodes and more
than one back-tracing paths exist.) To deal with arbitrary neural network
model, the calculation of receptive field region at each layer is conducted in
post-order Depth First Search with an intuition that $p$-layer receptive field
region can be represented as a function of receptive field regions of all the
child nodes of the $p$-layer node. In post-order Depth First Search, the
receptive field region of the $p$-layer will be calculated after all the child
nodes of the $p$-layer have been visited. The search algorithm ensures single
visit of each node and results in a worst-case complexity of
$\mathcal{O}(|\mathcal{E}|+|\mathcal{N}|)$, where $\mathcal{E}$ represents the
edge set and $\mathcal{N}$ represents the node set in the computation graph of
the model.

Recursive Case Setup: from p+1 to p. For simplicity of notation, we first
describe the problem setup of receptive field region calculation in
computation graph with only one path including all nodes. The setup can be
naturally extended to arbitrary computational graph with the Depth First
Search solution. Consider a pure convolutional neural network model with $P$
layer operations and the layer index $p\in[P]$. i.e, for any $p\in[P-1]$, the
node representing the $p$th layer in the computation graph has one and only
one child node representing the $(p+1)$th layer. The layer operation $type(p)$
can be one of the common operations in Neural Network including _Convolution,
Activation, Pooling, Normalization, Interpolation_. Define the feature map
$f_{p}$ as the output feature map of the $p$th layer. For any $p\in[P-1]$, We
want to derive a recursive and invertible mapping function that maps $r^{p+1}$
with respect to $f_{p+1}$ to $r^{p}$ with respect to $f_{p}$, based on the
type of the layer $(p+1)$ and a list of parameters $A_{p+1}$ that characterize
the layer $(p+1)$.

(1) $\displaystyle r^{p}=F(r^{p+1},A_{p+1})$

Recursive Case of Convolution. Under the single channel assumption mentioned
in previous section, 4 common parameters are left for the spatial
configuration of the $p$ th convolutional layer. Denote the kernel size as
$k_{p}$, stride as $s_{p}$, padding as $q_{p}$, dilation factor as $\alpha$.
i.e $A_{p}=[k_{p},s_{p},q_{p},\alpha]$. The function $F$ is deterministic and
unique, given the same state of the convolution operations. One representation
of the recursive mapping function F in 1-D dimension is given in (Araujo et
al., 2019). For 2D images, the derivations can be applied to both dimensions
independently because the receptive field region is rectangular in the
simplified problem setup. Denote $u_{p}$ and $v_{p}$ as the left-most and
right-most zero-indexed coordinates of $r_{p}$ with respect to $f_{p}$.

(2) $\displaystyle u_{p}=-q_{p+1}+u_{p+1}\cdot s_{p+1}$ (3) $\displaystyle
v_{p}=-q_{p+1}+v_{p+1}\cdot s_{p+1}+\alpha(k_{p+1}-1)$

$F$ is defined to map 1-D interval $[u_{p+1},v_{p+1}]$ to $[u_{p},v_{p}]$.
Note that the receptive field region can possibly exceed the boarder of the
feature map because of non-zero padding. In this case, the controller should
memorize how much it exceeds the boarder for a future Autopadding operation in
the forwarding pass and set $u_{p}=max(u_{p},0)$ and $v_{p}=min(v_{p},width)$.
The same operation is applied to two dimensions of images independently.

Recursive Case of Normalization. Batch normalization(Ioffe and Szegedy, 2015)
exists in many segmentation algorithms, as a widely-used and effective
normalization method. Batch normalization makes training of convolutional
neural network more stable through normalization along the batch dimension.
The batch normalization does not alter the receptive field region in
forwarding pass, which means the mapping function F to be an identity
function.

Recursive Case of Pooling and Interpolation. A pooling layer a fundamental
building block of a convolutional neural network, calculating value for
patches of a feature map. The pooling layer downsamples the feature map and
results in a larger receptive field. For receptive region computation purpose,
pooling by a factor of $x$ is equivalent to a convolutional layer with kernel
size and stride $x$. In contrast, upsampling is often done by interpolation
layer such as nearest neighbour, bilinear, bicubic and trilinear. Any pixel in
the output feature map of the interpolation layer depends locally on input
features. For receptive region computation purpose, the interpolation layer is
equivalent to a convolutional layer that has a kernel size equal to the number
of input features involved in the computation of an output feature.

Extended Setup for Arbitrary Computation Graph. Suppose the $p$-layer has m
children denoted as $p^{l}$ such that $l\in[m]$. Denote the receptive field
region $r^{p}$ with respect to the $p$th layer output $f_{p}$,

(4) $\displaystyle r^{p}=\bigcup_{l\in[m]}F(r^{p^{l}},A_{p^{l}})$

where $F$ denotes the recursive and invertible mapping function that maps
receptive field region of child nodes to the receptive field region with
respect to current feature map, based on the type of the child layer
operation. Note that the union of the outputs of multiple mapping function F
might be the union of more than one rectangular regions. In our PyTorch
implementation, we approximate the receptive field region $r_{p}$ by the
smallest rectangular region that fully covers all the back-traced rectangular
regions before we further compute the receptive field region at a higher
nodes. Note that the smallest rectangular region can be trivially computed by
simply comparing the coordinate values of sides of all rectangular receptive
field regions. The design choice is based on two practical reasons. 1)
Multiple back-traced rectangular regions are often greatly overlapped with
each other. Thus, negligible theoretical computation overheads are introduced
by computing features in the approximated receptive field region; 2) Pytorch
implementations are much more efficient in computing features in a rectangular
patch. At each layer, exact input and output of individual $F$ function are
memorized by the receptive field region controller for AutoCropping in the
forwarding pass.

AutoCropping and AutoPadding. After (approximate) layer receptive field
regions have been computed from leaf to root in the computation graph, we can
compute features within the approximate receptive field regions from root to
leaf. For any $p$ layer and any child node $p^{l}$ of the $p$ layer, the
receptive field region controller crops the feature map with the memorized
output of the $F$ function and pads it with the memorized padding value at
four boarders for next-layer feature computation in $p^{l}$. Any feature value
that is outside the approximate layer receptive field $r^{p}$ does not
contribute to final-layer features around the user click.

###  3.2. System Implementation Details

This section covers details of TraceNet system implementation besides the RFT
module. Note that the RFT module introduced above can be applied to
convolutional instance segmentation algorithms in general, with ”plug-in”
property to reduce computations and memory consumption.

Overall Design Choice. In addition to RFT, as shown in Figure 2, TraceNet
includes a feature pyramid backbone(Lin et al., 2017a) that extracts multi-
level feature maps from the input image, a conditional filter head that
predicts the parameters of the mask head with local features around the tap, a
compact mask head conditioned on the user-specified instance in their filters.
These components exist in different formats in most of the instance
segmentation algorithm paradigms that first attend to all the instances
exhaustedly in the image (step I) and then predict the pixel-level masks of
the instance (step II). (i.e instance first). Being universal to all of the
instance segmentation methods, the RFT can significantly prevent exhausted
computation over all the feature maps in step I by Receptive Field Region
Tracing mechanism, but cannot save computation in step II, as it is shown in
Figure 2. Note that conditional based methods like Condinst (Tian et al.,
2020) is different from and ROI operation based methods like Mask-RCNN (He et
al., 2017) in the sense that conditional based methods has heavy computation
in step I and a compact mask head in step II, while step II is much heavier in
ROI-operation based method. Thus by intuition, the RFT with conditional based
methods can result in more significant savings in conditional based methods
and less computation overheads left.

Backbone Module. Following the design of Feature Pyramid
Network (Lin et al., 2017a), we extract a 5-level feature pyramid over
Resnet(He et al., 2016). Each level of our feature pyramid is used to extract
local features around the tap at different scales. The feature pyramid design
is essential because no scaling information of the user-specified instance is
provided.

Mask Branch Conditioned on Instance. We follow the design of mask branch in
Condinst (Tian et al., 2020). The mask branch is in the format of Fully
Convolutional Neural Network (Long et al., 2015) for an image-level
prediction. The mask branch is applied to a feature map extracted from the
backbone. (i.e $P_{3}$ with downsampling ratio of 8). Compared to the mask
branch design in Mask RCNN (He et al., 2017), the design eliminates needs for
ROI operations by performing convolution in image-level. Besides, the
computation overhead of the mask head in our model is much more lightweight.
It only consists of 3 1x1 convolution layers with 8 channels each, while the
unconditioned mask branch in Mask-RCNN often has four convolution layers with
256 channels. The intuition behind the compact design can be explained as
follows. When the parameters of mask branch in our model are conditioned on
local features around user-specified instance, the characteristics (geometry
of the instance and relative location of the instance with respect to the user
query tap) of the user specified instance can be encoded in the mask branch
with the help of conditional filter head. To fully exploit the encoded spatial
characteristic, we concatenate the $P_{3}$ feature map with a map of relative
coordinates from all locations to the user tap query. Similar designs also
exist in other conditional instance segmentation algorithms (Tian et al.,
2020; Yu et al., 2022). When applied to the concatenated inputs, the mask
branch can naturally focus on the pixels of the user specified instance and
predict the mask in an instance-aware manner. A sigmoid is applied make the
mask prediction class-agnostic. Finally, a bilinear 4x upsampling is performed
on the output mask. The upsampling results in 2x downsampling mask prediction
compared to the resolution of the input image.

Conditional Filter Head and Box Head. The conditional filter head is adopted
with slight modification from Condinst(Tian et al., 2020), which is based on
FCOS(Tian et al., 2021). In FCOS and Condinst, each pixel location on multi-
level feature map can be associated with an instance in the original image by
a simple mapping. i.e. The pixel $(x,y)$ at the feature map with downsamling
ratio of $s$ can be mapped to the imput image as
$\left(\left\lfloor\frac{s}{2}\right\rfloor+xs,\left\lfloor\frac{s}{2}\right\rfloor+ys\right)$.
Condinst introduces a conditional filter head to encode characteristics of the
instance associated with the location in the feature map in pixel-level
fashion and a box head to regress the bounding box location of the same
instance. The conditional filter head is used to predict a 169-dimension
vector of parameters $\bm{\theta}_{a,b}$ for the above mentioned mask branch
for the mask of instance located at $(a,b)$ in the feature map. The box head
predicts a 4-dimension vector encoding relative distance between the pixel and
four boundaries of the bounding box. Both heads take the features extracted
from heavy classification and box tower, which consists of four 3x3 stride 1
convolutional layers with dimension 256 followed by ReLU activation and
BN(Ioffe and Szegedy, 2015). The centerness head is not included in the Figure
2 and we refer readers to more details in FCOS (Tian et al., 2021).

Training Target. The loss function adopted is similar to Condinst (Tian et
al., 2022):

(5) $\displaystyle
L=L_{\text{cls}}+\lambda_{1}L_{\text{mask}}+\lambda_{2}L_{\text{reg}}$

It is a balanced loss of classification, bounding box regression and mask
prediction. $\lambda_{1}$ and $\lambda_{2}$ are equal to 1 in the experiment.
We refer readers for more details of $L_{\text{cls}}$, $L_{\text{reg}}$ and
$L_{\text{mask}}$ to FCOS and Condinst. All of three losses are calculated and
averaged over all locations associated with arbitrary ground truth instance.

Training Implementation. We follow the training implementation of Condinst
(Tian et al., 2020). The model adopts Resnet-50, Resnet-101 (He et al., 2016)
with FPN (Lin et al., 2017a) as backbones. Weights in the backbone are
initialized with ImageNet pre-trained weights. Weights in other layers are
initialized as in (Lin et al., 2017b). The model is trained with SGD over 8
A100 GPUs for 90K iterations. Hyperparamter setting: Initial learning rate
0.01, mini-batch size 16, weight deacy 0.0001, momentum 0.9. The learning rate
decays by the factor of 0.1 at iteration 60K and 80K. At the beginning of the
training, a 1000-iteration warmup with fixed linear learning rate decay is
performed. The warmup factor is 0.001. Maintaining the aspect ratio, the input
images are resized to have the longer side less equal to 1333 and shorter side
in $[640,800]$ during training. Left-right flipping data augmentation is also
adopted in the training.

Inference. Given the tap query on the image, TraceNet first aligns the tap to
the five nearest location at each feature map, one per feature map. Then RFT
backtraces recursively all the receptive field regions across all layers with
respect to these five locations, such that all the calculation outside the
receptive field region does not contribute to the final predicted features at
these five locations. AutoCropping and Autopadding are performed during the
forwarding pass. The forwarding pass predicts $5\times C$ candidate foreground
bounding boxes/parameter vectors associated with the instance in total, where
$C$ indicates the category class of the dataset. NMS on bounding box with
threshold 0.6 is performed to select the top-1 result for producing the
instance mask at the tap.

##  4\. Experiments

Table 1. mIoU-T and mTA on LVIS, mTA on COCO. TraceNet highlights our method. Method | mIoU-T | mTA(LVIS) | mTA(COCO)
---|---|---|---
ritm-h32 (Sofiiuk et al., 2022) | 0.328 | 0.238 | 0.349
focuscut-R-50 (Lin et al., 2022) | 0.253 | 0.0734 | 0.117
focuscut-R-101 (Lin et al., 2022) | 0.228 | 0.0744 | 0.138
TraceNet-R-50-FPN | 0.286 | 0.272 | 0.346
TraceNet-R-101-FPN | 0.294 | 0.257 | 0.395
Table 2. mIoU-T over categories on COCO dataset. Method | person | car | chair | bottle | cup | dining table | traffic light | bowl | Category Total
---|---|---|---|---|---|---|---|---|---
mobilesam (Zhang et al., 2023) | 0.4877 | 0.5129 | 0.3530 | 0.6043 | 0.6405 | 0.2571 | 0.3425 | 0.3675 | 0.4331
focuscut-R-50 (Lin et al., 2022) | 0.4128 | 0.3734 | 0.3703 | 0.3705 | 0.4715 | 0.2206 | 0.3652 | 0.4631 | 0.3774
focuscut-R-101 (Lin et al., 2022) | 0.4759 | 0.4150 | 0.3961 | 0.3079 | 0.3610 | 0.2735 | 0.3122 | 0.4411 | 0.4217
TraceNet-R-50-FPN | 0.4306 | 0.4796 | 0.3285 | 0.090 | 0.1631 | 0.5010 | 0.2925 | 0.2541 | 0.3977
TraceNet-R-101-FPN | 0.4542 | 0.5186 | 0.3787 | 0.0958 | 0.1798 | 0.5207 | 0.4471 | 0.2733 | 0.4312
Table 3. Computation Cost to Retrieve an Instance from an image of size of 1024 x 768. TraceNet highlights our method. FullNet refers to our model without RFT. Method | throughput | latency (ms) | FLOPs)
---|---|---|---
mobilesam (Zhang et al., 2023) | 40.81 | 24.50 | -
ritm-h32 (Sofiiuk et al., 2022) | 11.37 | 87.94 | 406.5G
focuscut-R-50 (Lin et al., 2022) | 44.33 | 22.56 | 52.55G
focuscut-R-101 (Lin et al., 2022) | 37.86 | 26.42 | 67.18G
FullNet-R-50-FPN | - | - | 86.67G
FullNet-R-101-FPN | - | - | 118.28G
TraceNet-R-50-FPN | 48.85 | 20.47 | 34.47G
TraceNet-R-101-FPN | 41.96 | 23.83 | 66.67G

###  4.1. Evaluation Formulation

Denote a set of ground-truth masks $\left\\{M_{gt}^{i}\right\\}$, where
$M_{gt}^{i}$ is the ground-truth mask of the $i$-th instance in the dataset.
Denote a set of taps $\left\\{c^{i,j}\right\\}$ where each element $c^{i,j}$
is the $j$-th tap, $c^{i,j}\in M_{gt}^{i}$. Denote a set of predicted masks as
$\left\\{M_{pred}^{i,j}\right\\}$, where $M_{pred}^{i,j}$ is the predicted
mask of the $i$-th instance when being queried by $c^{i,j}$.

###  4.2. Evaluation Protocol

We propose two metrics for tap-driven segmentation task, mean Tap Intersection
over Union (mIoU-T) and mean Tap Area (mTA) as discussed in Section 3.

Mean Tap Intersection Over Union. Since the predicted mask $M_{pred}$ depends
on the user tap, IoU cannot be directly applied to measure the expectation of
mask quality with different user taps. We propose mIoU-T that measures the
average quality of a set of predicted masks $\left\\{M_{pred}^{i,j}\right\\}$
over a set of instances with ground-truth mask $M_{gt}^{i}$.

(6) $\displaystyle
mIoU_{T}=\frac{\sum_{i,j}\text{Area}(M^{i,j}_{\text{pred}}\cap
M^{i}_{\text{gt}})}{\sum_{i,j}\text{Area}(M^{i,j}_{\text{pred}}\cup
M^{i}_{\text{gt}})}$

This is a general metric that works on arbitrary numbers of one-shot tap
queries and arbitrary numbers of instances. In our experiment, the number of
taps is a constant within each instance. i.e for any $i$, $j\in[k]$, where k
is a constant value indicating the size of the set of candidate one-shot tap
queries for an instance.

Mean Tap Area. Besides the expectation of predicted mask quality, the user tap
tolerance is measured, i.e., the proportion of the region that a user tap can
fall into for a high-quality single-instance mask. We propose to use mTA to
measure user tap tolerance. To calculate the feasible region of user tap, for
any $i$, $\left\\{c^{i,j}\right\\}$ should be constructed as a set of all
pixels in the $M_{gt}^{i}$. mTA is calculated as below:

(7) $\displaystyle
mTA=\frac{\sum_{i,j}\mathds{1}(\text{IoU}(M^{i,j}_{\text{pred}},M^{i}_{\text{gt}})\geq\beta)}{\sum_{i}\text{Area}(M^{i}_{\text{gt}})}$

where $\beta$ is a pre-defined IoU threshold and $\mathbb{1}$ is an indicator
function. We use Eq. 4.2 rather than averaging among each of the trials to
treat instances of various sizes equally.

###  4.3. Implementation Detail

Dataset We train and evaluate TraceNet on MS-COCO (Lin et al., 2014) and LVIS
(Gupta et al., 2019). MS-COCO is a large-scale dataset for object detection
and instance segmentation with over 82k training images and 600k instance-
level mask annotations. LVIS is a dataset built from the images in MS-COCO for
large vocabulary instance segmentation. It has 2-million high-quality instance
mask annotations over 1k entry-level categories.

Figure 4. Generated tap regions from band-1 (left most) to band-5 (right
most).

Tap Simulation with Morphological Transformations. Since no taps are provided
in the dataset, we have to simulate user taps within all instances in the
dataset. During evaluation, 25 user taps $\left\\{c^{i,j}\right\\}$ are
simulated within all instances in the dataset. We sample 25 taps for one
instance, with each five taps randomly sampled in one of the five bands around
the moment of the instance. The boundary between bands is constructed with the
help of morphological transformations, as is described as followings: Suppose
the instance is bounded by a bounding box in ${H\times W}$. Then we construct
an binary image $I\in\\{0,1\\}^{H\times W}$, where only the pixel value at
moment of the instance is one. The first boundary between bands is generated
by performing dilation on the $I$ with kernel size ${H/5\times W/5}$. More
boundaries are generated by performing the same dilation operation repeatedly.
The generated taps are visualized in the Figure 4.

Figure 5. mIoU-T and mTA over different band regions in COCO, LVIS and BDD100K
(additional) from left to right Figure 6. Qualitative Results: first column as
groundtruth.

###  4.4. Quantitative Evaluation

Baselines. ritm (Sofiiuk et al., 2022) is a computationally heavy click-based
interactive segmentation model with the strongest one-click performance before
SAM (Kirillov et al., 2023). focuscut (Lin et al., 2022) is a state-of-the-art
efficient click-based interactive segmentation model with the ResNet backbone.
For a fair comparison, focuscut, ritm, and our models are trained on LVIS +
COCO. Mobilesam is trained on a significantly larger dataset proposed in the
SAM paper.

mIoU-T. We evaluate the overall IoU-T performance of our method on COCO and
LVIS. As shown in Table 1 and Table 2, the TraceNet with ResNet backbone
outperforms focuscut with the same backbone. The performance is slightly lower
than heavy ritm and mobilesam with more training data. In the category
analysis, our model achieves good performance on common and large object
categories (e.g. dining table and car) but loses performance in small ones
(e.g. bottle and cup) compared to baselines. The intuition is that global
features eliminated by RFT module are important when extracting segmentation
masks of relatively small objects. However, the property of performing better
in large salient objects than small objects are favoured in mobile
applications, since people intend to zoom in to click on the instance in real-
time editing or capturing.

mTA. mTAs of TraceNet on COCO are 0.346 and 0.395 with ResNet-50 and
RestNet-101 backbone, respectively. It indicates that over 30$\%$ of the user
taps can result in a good instance mask with IoU larger than 0.7. A
significant drop in terms of mTA is observed on LVIS. Since the LVIS dataset
provides more fine-grained annotations of the image in COCO. Overall, TraceNet
achieves significantly better mTA than baselines, demonstrating robustness to
tap locations.

Performance Profiling over Band The mIoU-T and mTA performance over 5
different bands are profiled in Figure 5. As it is shown in the figure, both
mIoU-T and mTA drop with the distance to the instance center. As it is shown
in the second row of Figure 5, a good instance mask could be obtained within
the Band-1 and Band-2. The quality of the instance mask will drop
significantly when a user tap to a region is far from the instance center.

Table 4. Computation Cost of other non-efficient baselines in FLOPs. Model_backbone | Size of Input | FLOPs
---|---|---
Iseg_SwinB (Liu et al., 2022a) | 400 x 400 | 138.21
Iseg_SwinL (Liu et al., 2022a) | 400 x 400 | 302.78
FocalClick_SegFB3 (Chen et al., 2022) | 256 x 256 | 24.75
CDNet_R-34 (Chen et al., 2021) | 384 x 384 | 113.60
SimpleClick_ViT-Tiny (Liu et al., 2023) | 224 x 224 | 2.63
SimpleClick_ViT-Tiny (Liu et al., 2023) | 448 x 448 | 10.52
SimpleClick_ViT-H (Liu et al., 2023) | 224 x 224 | 532.87
SimpleClick_ViT-H (Liu et al., 2023) | 448 x 448 | 1401.93

Computation analysis. Efficiency is one of the most important objectives of
one-tap segmentation task for mobile devices. In Table 3, we make an analysis
on FLOPs, throughput in one second, and latency of the inference process of
the algorithm. With the help of RFT, TraceNet saves 60.2% of computations on
R-50-FPN Backbone and 43.6% of computations on R-101-FPN, compared to the full
inference process which is denoted as FullNet (for ablation). TraceNet
significantly outperforms all baselines in terms of computation efficiency. To
show efficiency superiority of Tracenet, we also include table 4 showing
extremely high flops of non-efficient interactive segmentation algorithms on
input images of much smaller resolution. Most of them are transformer based
models. The FLOPs of a transformer block scales quickly with increase in input
resolution. It can be calculated as $12nd^{2}+2n^{2}d$. (Pan et al., 2021).
$n$ denotes the input sequence length and d denotes the embedding dimension.
Large Model such as SAM is even more expensive than the above mentioned models
and takes 0.15s per frame on Nvidia A100 GPU. In terms of memory consumption,
TraceNet, with its minimal requirement of only 12.5M parameters for inference,
is optimally designed to meet the demands of mobile devices.

###  4.5. Qualitative Result Evaluation

The qualitative evaluation of our TraceNet method is depicted in Figure 6. In
the first column of each image set, we present the comprehensive ground-truth
segmentation masks that encompass all dataset categories represented within
each single image. The subsequent columns showcase the results of instance
masks generated from user interactions, where users have performed a random
tap, selecting one instance at a time. The displayed results demonstrate the
exceptional performance of the TraceNet in producing instance masks.
Remarkably, the model requires only a single user tap per instance to predict
the masks with high precision. This level of accuracy is mostly maintained
even when the tap is placed on the non-centered regions of an instance. This
robustness of TraceNet demonstrates the model’s excellent ability to segment
the full extent of an instance from minimal and potentially suboptimal input.
Such capabilities are indicative of model’s advances understanding of the
contextual information at all levels within the images, making it an ideal
candidate for camera applications that benefit from on-the-fly photo editing
features, where users may prefer quick and intuitive interactions with the
editing tool.

##  5\. Conclusion

In this paper, we propose and formulate one-tap efficient segmentation task.
We also design the evaluation protocol including mIoU-T and mTA metrics. We
build our solution TraceNet that back-traces the receptive field region at
each layer with respect to local features around the user query tap. TraceNet
demonstrates effectiveness and efficiency in the inference process,
potentially benefiting on-the-fly mobile image editing applications. TraceNet
serves as a practical efficient and interactive solution on mobile before we
achieve extremely fast inference of large promptable models. Besides, the
success of TraceNet shows potentials to leverage one ”referring” modality to
significantly increase efficiency.

## References

  * (1)
  * Amazon (2023) Amazon. 2023.  _Amazon Flows_.  https://www.amazon.com/A9-Innovations-LLC-Powered-Amazon/dp/B008G318PE
  * Araujo et al. (2019) André Araujo, Wade Norris, and Jack Sim. 2019.  Computing Receptive Fields of Convolutional Neural Networks.  _Distill_ (2019).  https://doi.org/10.23915/distill.00021 https://distill.pub/2019/computing-receptive-fields.
  * Bolya et al. (2020a) Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. 2020a.  YOLACT++: Better Real-time Instance Segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2020).
  * Bolya et al. (2020b) Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. 2020b.  YOLACT++: Better Real-time Instance Segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2020).
  * Boykov and Jolly (2001) Y.Y. Boykov and M.-P. Jolly. 2001.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images. In _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , Vol. 1. 105–112 vol.1.  https://doi.org/10.1109/ICCV.2001.937505
  * Cao et al. (2020) Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao. 2020.  SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation.  _Proc. European Conference on Computer Vision_ (2020).
  * Chen et al. (2018) Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. In _ECCV_.
  * Chen et al. (2021) Xi Chen, Zhiyan Zhao, Feiwu Yu, Yilei Zhang, and Manni Duan. 2021.  Conditional Diffusion for Interactive Segmentation. In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. 7325–7334.  https://doi.org/10.1109/ICCV48922.2021.00725
  * Chen et al. (2022) Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. 2022.  FocalClick: Towards Practical Interactive Image Segmentation. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_. IEEE, 1290–1299.  https://doi.org/10.1109/CVPR52688.2022.00136
  * Cheng et al. (2022) Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. 2022.  Masked-attention Mask Transformer for Universal Image Segmentation.  _CVPR_.
  * Grady (2006) L. Grady. 2006.  Random Walks for Image Segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 28, 11 (2006), 1768–1783.  https://doi.org/10.1109/TPAMI.2006.233
  * Gupta et al. (2019) Agrim Gupta, Piotr Dollar, and Ross Girshick. 2019.  Lvis: A dataset for large vocabulary instance segmentation. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 5356–5364.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017.  Mask R-CNN. In _2017 IEEE International Conference on Computer Vision (ICCV)_. 2980–2988.  https://doi.org/10.1109/ICCV.2017.322
  * He et al. (2016) Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep Residual Learning for Image Recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2016), 770–778.
  * Howard et al. (2019) Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. 2019.  Searching for MobileNetV3. In _ICCV_.
  * Hu et al. (2021) Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji. 2021.  Istr: End-to-end instance segmentation with transformers.  _arXiv preprint arXiv:2105.00637_ (2021).
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In _Proceedings of the 32nd International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 37)_ , Francis Bach and David Blei (Eds.). PMLR, Lille, France, 448–456.  https://proceedings.mlr.press/v37/ioffe15.html
  * Jang and Kim (2019) Won-Dong Jang and Chang-Su Kim. 2019.  Interactive Image Segmentation via Backpropagating Refinement Scheme. In _Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition_.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. 2023.  Segment Anything.  _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ (2023), 3992–4003.  https://api.semanticscholar.org/CorpusID:257952310
  * Lee and Park (2020) Youngwan Lee and Jongyoul Park. 2020.  CenterMask: Real-Time Anchor-Free Instance Segmentation.  (2020).
  * Lin et al. (2017a) Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017a.  Feature Pyramid Networks for Object Detection. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 936–944.  https://doi.org/10.1109/CVPR.2017.106
  * Lin et al. (2017b) Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2017b.  Focal Loss for Dense Object Detection.  _2017 IEEE International Conference on Computer Vision (ICCV)_ (2017), 2999–3007.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _European conference on computer vision_. Springer, 740–755.
  * Lin et al. (2022) Zheng Lin, Zheng-Peng Duan, Zhao Zhang, Chun-Le Guo, and Ming-Ming Cheng. 2022.  FocusCut: Diving into a Focus View in Interactive Segmentation. In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 2627–2636.  https://doi.org/10.1109/CVPR52688.2022.00266
  * Lin et al. (2020) Zheng Lin, Zhao Zhang, Lin-Zhuo Chen, Ming-Ming Cheng, and Shao-Ping Lu. 2020.  Interactive Image Segmentation With First Click Attention. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 13336–13345.  https://doi.org/10.1109/CVPR42600.2020.01335
  * Liu et al. (2023) Qin Liu, Zhenlin Xu, Gedas Bertasius, and Marc Niethammer. 2023.  SimpleClick: Interactive Image Segmentation with Simple Vision Transformers.  arXiv:2210.11006 [cs.CV]
  * Liu et al. (2022a) Qin Liu, Zhenlin Xu, Yining Jiao, and Marc Niethammer. 2022a.  ISegFormer: Interactive Segmentation Via Transformers With Application To 3D Knee MR Images. In _Medical Image Computing and Computer Assisted Intervention – MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part V_ (Singapore, Singapore). Springer-Verlag, Berlin, Heidelberg, 464–474.  https://doi.org/10.1007/978-3-031-16443-9_45
  * Liu et al. (2022b) Qin Liu, Meng Zheng, Benjamin Planche, Srikrishna Karanam, Terrence Chen, Marc Niethammer, and Ziyan Wu. 2022b.  PseudoClick: Interactive Image Segmentation with Click Imitation.  https://doi.org/10.48550/ARXIV.2207.05282
  * LLC (2023) Google LLC. 2023.  _Google Lens_.  https://blog.google/products/google-lens/google-lens-features/
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015.  Fully convolutional networks for semantic segmentation. In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 3431–3440.  https://doi.org/10.1109/CVPR.2015.7298965
  * Luo et al. (2016) Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. 2016.  Understanding the Effective Receptive Field in Deep Convolutional Neural Networks. In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ (Barcelona, Spain) _(NIPS’16)_. Curran Associates Inc., Red Hook, NY, USA, 4905–4913.
  * Microsoft (2020) Microsoft. 2020.  _Microsoft Seeing AI_.  https://www.microsoft.com/en-us/garage/wall-of-fame/seeing-ai/
  * Mortensen and Barrett (1995) Eric N. Mortensen and William A. Barrett. 1995.  Intelligent Scissors for Image Composition. In _Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’95)_. Association for Computing Machinery, New York, NY, USA, 191–198.  https://doi.org/10.1145/218380.218442
  * Mortensen and Barrett (1998) Eric N. Mortensen and William A. Barrett. 1998.  Interactive Segmentation with Intelligent Scissors.  _Graph. Models Image Process._ 60, 5 (sep 1998), 349–384.  https://doi.org/10.1006/gmip.1998.0480
  * Najibi et al. (2019) Mahyar Najibi, Bharat Singh, and Larry S Davis. 2019.  AutoFocus: Efficient Multi-Scale Inference.  _ICCV_ (2019).
  * Pan et al. (2021) Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai. 2021.  Scalable Vision Transformers With Hierarchical Pooling. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 377–386.
  * Rother et al. (2004) Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. 2004.  ”GrabCut”: Interactive Foreground Extraction Using Iterated Graph Cuts.  _ACM Trans. Graph._ 23, 3 (aug 2004), 309–314.  https://doi.org/10.1145/1015706.1015720
  * Sandler et al. (2018) Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018.  MobileNetV2: Inverted Residuals and Linear Bottlenecks. In _CVPR_.
  * Sofiiuk et al. (2020) Konstantin Sofiiuk, Ilia Petrov, Olga Barinova, and Anton Konushin. 2020.  f-brs: Rethinking backpropagating refinement for interactive segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8623–8632.
  * Sofiiuk et al. (2022) Konstantin Sofiiuk, Ilya A. Petrov, and Anton Konushin. 2022.  Reviving Iterative Training with Mask Guidance for Interactive Segmentation. In _2022 IEEE International Conference on Image Processing (ICIP)_. 3141–3145.  https://doi.org/10.1109/ICIP46576.2022.9897365
  * Tian et al. (2020) Zhi Tian, Chunhua Shen, and Hao Chen. 2020.  Conditional Convolutions for Instance Segmentation. In _Proc. Eur. Conf. Computer Vision (ECCV)_.
  * Tian et al. (2021) Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. 2021.  FCOS: A Simple and Strong Anchor-free Object Detector.  (2021).
  * Tian et al. (2022) Zhi Tian, Bowen Zhang, Hao Chen, and Chunhua Shen. 2022.  Instance and Panoptic Segmentation Using Conditional Convolutions.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2022), 1–1.  https://doi.org/10.1109/TPAMI.2022.3145407
  * Wang et al. (2020) Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. 2020.  SOLOv2: Dynamic and Fast Instance Segmentation.  _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ (2020).
  * Xie et al. (2019) Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. 2019.  PolarMask: Single Shot Instance Segmentation with Polar Representation.  _arXiv preprint arXiv:1909.13226_ (2019).
  * Xu et al. (2016) Ning Xu, Brian L. Price, Scott Cohen, Jimei Yang, and Thomas S. Huang. 2016.  Deep Interactive Object Selection. In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_. IEEE Computer Society, 373–381.  https://doi.org/10.1109/CVPR.2016.47
  * Yang et al. (2021) Chenhongyi Yang, Zehao Huang, and Naiyan Wang. 2021.  QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection.  _arXiv preprint arXiv:2103.09136_ (2021).
  * Yu et al. (2022) Xiaodong Yu, Dahu Shi, Xing Wei, Ye Ren, Tingqun Ye, and Wenming Tan. 2022.  SOIT: Segmenting Objects with Instance-Aware Transformers. In _Proceedings of the AAAI Conference on Artificial Intelligence_. 3188–3196.
  * Zhang et al. (2023) Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. 2023.  Faster Segment Anything: Towards Lightweight SAM for Mobile Applications.  _arXiv preprint arXiv:2306.14289_ (2023).
  * Zhou et al. (2023) Minghao Zhou, Hong Wang, Qian Zhao, Yuexiang Li, Yawen Huang, Deyu Meng, and Yefeng Zheng. 2023.  Interactive Segmentation as Gaussian Process Classification.  arXiv:2302.14578 [cs.CV]
  * Zou et al. (2023) Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. 2023.  Segment Everything Everywhere All at Once. In _Thirty-seventh Conference on Neural Information Processing Systems_.  https://openreview.net/forum?id=UHBrWeFWlL