# PolyFormer: Referring Image Segmentation as Sequential Polygon Generation

Jiang Liu1,†, Hui Ding2,† Zhaowei Cai2 Yuting Zhang2
Ravi Kumar Satzoda2 Vijay Mahadevan2 R. Manmatha2
Johns Hopkins University1 AWS AI Labs2
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work done
during internship at AWS AI Labs. ${\dagger}$ Equal contribution.

###### Abstract

In this work, instead of directly predicting the pixel-level segmentation
masks, the problem of referring image segmentation is formulated as sequential
polygon generation, and the predicted polygons can be later converted into
segmentation masks. This is enabled by a new sequence-to-sequence framework,
Polygon Transformer (PolyFormer), which takes a sequence of image patches and
text query tokens as input, and outputs a sequence of polygon vertices
autoregressively. For more accurate geometric localization, we propose a
regression-based decoder, which predicts the precise floating-point
coordinates directly, without any coordinate quantization error. In the
experiments, PolyFormer outperforms the prior art by a clear margin, e.g.,
5.40% and 4.52% absolute improvements on the challenging RefCOCO+ and RefCOCOg
datasets. It also shows strong generalization ability when evaluated on the
referring video segmentation task without fine-tuning, e.g., achieving
competitive 61.5% $\mathcal{J}\&\mathcal{F}$ on the Ref-DAVIS17 dataset.

##  1 Introduction

Referring image segmentation (RIS) [48, 43, 59, 72, 7, 86, 29, 31, 30, 20, 19,
85, 77, 37, 28] combines vision-language understanding [53, 74, 55, 91, 40]
and instance segmentation [24, 50, 8, 16, 2], and aims to localize the
segmentation mask of an object given a natural language query. It generalizes
traditional object segmentation from a fixed number of predefined categories
to any concept described by free-form language, which requires a deeper
understanding of the image and language semantics. The conventional pipeline
[48, 43, 59, 72, 7, 86, 29, 31, 30, 20, 19, 28] first extracts features from
the image and text inputs, and then fuses the multi-modal features together to
predict the mask.

A segmentation mask encodes the spatial layout of an object, and most instance
segmentation models [24, 50, 8, 16] rely on a dense binary classification
network to determine whether each pixel belongs to the object. This pixel-to-
pixel prediction is preferred by a convolutional operation, but it neglects
the structure among the output predictions. For example, each pixel is
predicted independently of other pixels. In contrast, a segmentation mask can
also be represented by a sparse set of structured polygon vertices delineating
the contour of the object [47, 80, 6, 45, 1, 39]. This structured sparse
representation is cheaper than a dense mask representation and is the
preferred annotation format for most instance segmentation datasets [47, 15,
69]. Thus, it is also tempting to predict structured polygons directly.
However, how to effectively predict this type of structured outputs is
challenging, especially for convolutional neural networks (CNNs), and previous
efforts have not shown much success yet [80, 45, 39].

Figure 1:  The illustration of PolyFormer pipeline for referring image
segmentation (polygon vertex sequence) and referring expression comprehension
(bounding box corner points). The polygons is converted to segmentation masks
in the end.

We address this challenge by resorting to a sequence-to-sequence (seq2seq)
framework [73, 14, 64, 65, 3], and propose Polygon transFormer (PolyFormer)
for referring image segmentation. As illustrated in Fig. 1, it takes a
sequence of image patches and text query tokens as input, and autoregressively
outputs a sequence of polygon vertices. Since each vertex prediction is
conditioned on all preceding predicted vertices, the output predictions are no
longer independent of each other. The seq2seq framework is flexible on its
input and output format, as long as both of them can be formulated as
sequences of variable length. Thus, it is natural to concatenate the visual
and language features together as a long sequence, avoiding complicated multi-
modal feature fusion as in prior work [72, 7, 86, 29, 85]. In the meantime,
the output can also be a long sequence of multiple polygons separated by
separator tokens, covering the scenario where the segmentation masks are not
connected, _e.g_., by occlusion, as shown in Fig. 1. Furthermore, since a
bounding box can be represented as a sequence of two corner points (_i.e_.,
top left and bottom right), they can also be output by PolyFormer along with
the polygon vertices. Thus, referring image segmentation (polygon) and
referring expression comprehension (bounding box) can be unified in our simple
PolyFormer framework.

Localization is important for polygon and bounding box generation, as a single
coordinate prediction mistake may result in substantial errors in the mask or
bounding box prediction. However, in recent seq2seq models for the visual
domain [9, 10, 76, 54], in order to accommodate all tasks in a unified seq2seq
framework, the coordinates are quantized into discrete bins and the prediction
is formulated as a classification task. This is not ideal since geometric
coordinates lie in a continuous space instead of a discrete one, and
classification is thus usually suboptimal for localization task [22, 41, 92].
Instead, we formulate localization as a regression task, due to its success in
object detection [23, 67, 24, 5], where floating-point coordinates are
directly predicted without any quantization error. Motivated by [24], the
feature embedding for any floating-point coordinate in PolyFormer is obtained
by bilinear interpolation [32] of its neighboring indexed embeddings. This is
in contrast with the common practice [9, 76, 54] in which coordinate feature
is indexed from a dictionary with a fixed number of discrete coordinate bins.
These changes enable our PolyFormer to make accurate polygon and bounding box
predictions.

We evaluate PolyFormer on three major referring image segmentation benchmarks.
It achieves 76.94%, 72.15%, and 71.15% mIoU on the validation sets of RefCOCO
[88], RefCOCO+ [88] and RefCOCOg [58], outperforming the state of the art by
absolute margins of 2.48%, 5.40%, and 4.52%, respectively. PolyFormer also
shows strong generalization ability when directly applied to the referring
video segmentation task without finetuning. It achieves 61.5%
$\mathcal{J}\&\mathcal{F}$ on the Ref-DAVIS17 dataset [36], comparable with
[79] which is specifically designed for that task.

Our main contributions are summarized as follows:

  * •

We introduce a novel framework for RIS and REC, called PolyFormer, which
formulates them as a sequence-to-sequence prediction problem. Due to its
flexibility, it can naturally fuse multi-modal features together as input and
generate a sequence of polygon vertices and bounding box corner points.

  * •

We propose a regression-based decoder for accurate coordinate prediction in
this seq2seq framework, which outputs continuous 2D coordinates directly
without quantization error. To the best of our knowledge, this is the first
work formulating geometric localization as a regression task in seq2seq
framework instead of classification as in [9, 10, 76, 54].

  * •

For the first time, we show that the polygon-based method surpasses mask-based
ones across all three main referring image segmentation benchmarks, and it can
also generalize well to unseen scenarios, including video and synthetic data.

Figure 2:  Overview of our PolyFormer architecture. The model takes an image
and its corresponding language expression as input, and outputs the floating-
point 2D coordinates of bounding box and polygons in an autoregressive way.

##  2 Related Work

#### Referring Image Segmentation

(RIS) [28] aims to provide pixel-level localization of a target object in an
image described by a referring expression. The previous works mainly focus on
two aspects: (1) vision and language feature extraction, and (2) multi-modal
feature fusion. For feature extraction, there has been a rich line of work,
including the use of CNNs [28, 48, 87, 86, 12, 7, 29, 30, 31], recurrent
neural networks [70, 12, 87, 13, 57, 20], and transformer models [37, 85, 42].
The efforts on feature fusion have explored feature concatenation [28, 48],
attention mechanisms [72, 7, 86, 29], and multi-modal Transformers [19, 77,
42, 37]. The method most related to ours is SeqTR [93], which also adopts a
transformer model for generating the polygon vertices sequentially. However,
SeqTR can only produce a single polygon of 18 vertices with coarse
segmentation mask, failing to outline objects with complex shapes and
occlusion.

#### Referring Expression Comprehension

(REC) predicts a bounding box that tightly encompasses the target object in an
image corresponding to a referring expression. Existing works include two-
staged methods [27, 90, 94, 26] that are based on region proposal ranking, and
one-stage methods [34, 42, 93, 46, 84, 4] that directly predict the target
bounding box. Several papers [42, 93, 57] explore multi-task learning of REC
and RIS since they are two closely related tasks. However, MCN [57] and RefTR
[42] require task-specific heads. Although SeqTR [93] casts the two tasks as a
point prediction problem in a unified framework, it shows that multi-task
supervision degenerates the performance compared with the single-task variant.
In contrast, our PolyFormer achieves improved performance via multi-task
learning of RIS and REC.

#### Sequence-to-Sequence (seq2seq) Modeling

has achieved a lot of successes in natural language processing (NLP) [73, 14,
64, 65, 3]. Sutskever _et al_. [73] propose a pioneering seq2seq model based
on LSTM [70] for machine translation. Raffel _et al_. [65] come up with the T5
model to unify various tasks including translation, question answering and
classification in a text-to-text framework. [3] further shows that scaling up
language models greatly improves few-shot performance. Inspired by these
successes in NLP, recent endeavors in computer vision and vision-language also
start to explore seq2seq modeling for various tasks [9, 10, 83, 76, 54, 93].
However, they cast geometric localization tasks as a classification problem,
_i.e_., quantizing coordinates into discrete bins and predicting the
coordinates as one of the bins. This enables them to unify all tasks into a
simple unified seq2seq framework, but neglects the differences among tasks. In
PolyFormer, the geometric localization is formulated as a more suitable
regression task that predicts continuous coordinates without quantization.

#### Contour-based Instance Segmentation

aims to segment instances by predicting the contour. [6] labels object
instances with polygons via a recurrent neural network. PolarMask [80] models
instance masks in polar coordinates, and converts instance segmentation to
instance center classification and dense distance regression tasks. Deep Snake
[61] extends the classic snake algorithm [81] and uses a neural network to
iteratively deform an initial contour to match the object boundary.
PolyTransform [45] exploits a segmentation network to first generate instance
masks to initialize polygons, which are then fed into a deformation network to
better fit the object boundary. BoundaryFormer [39] introduces a point-based
transformer with mask supervision via a differentiable rasterizer. However,
these papers are limited in how they handle fragmented objects.

##  3 PolyFormer

###  3.1 Architecture Overview

Fig. 2 gives an overview of PolyFormer architecture. Instead of predicting
dense segmentation masks, PolyFormer sequentially produces the corner points
of bounding box and vertices of polygons outlining the object. Specifically,
we first use a visual encoder and a text encoder to extract image and text
features, respectively, which are then projected into a shared embedding
space. Next, we concatenate the image and text features, and feed them into a
multi-modal transformer encoder. Finally, a regression-based transformer
decoder takes the encoded features and outputs the continuous floating-point
bounding box corner points and polygon vertices in an autoregressive way. The
segmentation mask is generated as the region encompassed by the polygons.

###  3.2 Target Sequence Construction

We first describe how to represent ground-truth bounding box and polygon
sequences.

#### Polygon Representation.

A segmentation mask is described using one or more polygons outlining the
referred object. We parameterize a polygon as a sequence of 2D vertices
$\\{(x_{i},y_{i})\\}_{i=1}^{K}$ , $(x_{i},y_{i})\in\mathbb{R}^{2}$ in the
clock-wise order. We choose the vertex that is closest to the top left corner
of the image as the starting point of the sequence, as shown in Fig. 3.

#### Vertex and Special Tokens.

For each vertex coordinate $x$ or $y$, previous works [9, 10, 76, 54, 83, 93]
uniformly quantize it into an integer between $[1,B]$, where $B\in\mathbb{N}$
is the number of bins of the coordinate codebook. In contrast, we maintain the
continuous floating-point value of the original $x$ or $y$ coordinate without
any quantization. To represent multiple polygons, we introduce a separator
token <SEP> between two polygons. Polygons from the same object are ordered
based on the distance between their starting points and the image origin.
Finally, we use <BOS> and <EOS> tokens to indicate the beginning and end of
the sequence.

#### Unified Sequence with Bounding Box.

A bounding box is represented by two corner points, _i.e_., top-left
$(x_{1}^{b},y_{1}^{b})$ and bottom-right $(x_{2}^{b},y_{2}^{b})$. The
coordinates of bounding box and multiple polygons can be concatenated together
into a single long sequence as follows:

$\displaystyle[$
$\displaystyle\texttt{<BOS>},(x_{1}^{b},y_{1}^{b}),(x_{2}^{b},y_{2}^{b}),(x_{1}^{1},y_{1}^{1}),$
$\displaystyle(x_{2}^{1},y_{2}^{1}),...,\texttt{<SEP>},(x_{1}^{n},y_{1}^{n}),...,\texttt{<EOS>}],$

where $(x_{1}^{n},y_{1}^{n})$ is the starting vertex of the $n^{th}$ polygon.
In general, the bounding box corner points and polygon vertices are regarded
as the coordinate tokens <COO>.

Figure 3:  Illustration of polygon sequence representation. The vertices in a
polygon are sorted in the clock-wise order, where the starting points (orange
dots) are the vertices that are closest to the image origin. The segmentation
mask is generated as the region encompassed by the polygons.

###  3.3 Image and Text Feature Extraction

As illustrated in Fig. 2, the input of our framework consists of an image $I$
and a referring expression $T$.

#### Image Encoder.

For an input image $I\in\mathbb{R}^{H\times W\times 3}$, we use a Swin
transformer [51] to extract the feature from the 4-th stage as visual
representation $F_{v}\in\mathbb{R}^{\frac{H}{32}\times\frac{W}{32}\times
C_{v}}$.

#### Text Encoder.

Given the language description $T\in\mathbb{R}^{L}$ with $L$ words, we use the
language embedding model from BERT [18] to extract the word feature
$F_{l}\in\mathbb{R}^{L\times C_{l}}$.

#### Multi-modal Transformer Encoder.

To fuse the image and textual features, we flatten $F_{v}$ into a sequence of
visual features
$F^{\prime}_{v}\in\mathbb{R}^{(\frac{H}{32}\cdot\frac{W}{32})\times C_{v}}$
and project $F^{\prime}_{v}$ and $F_{l}$ into the same embedding space with a
fully-connected layer:

$F^{\prime}_{v}=F^{\prime}_{v}W_{v}+b_{v},\ F^{\prime}_{l}=F_{l}W_{l}+b_{l},$
(1)

where $W_{v}$ and $W_{l}$ are learnable matrices to transform the visual and
textual representations into the same feature dimension, $b_{v}$ and $b_{l}$
are the bias vectors. The projected image and text features are then
concatenated: $F_{M}=[F^{\prime}_{v},F^{\prime}_{l}].$

The multi-modal encoder is composed of $N$ transformer layers, where each
layer consists of a multi-head self-attention layer, a layer normalization and
a feed-forward network. It takes the concatenated feature $F_{M}$ and
generates the multi-modal feature $F_{M}^{N}$ progressively.

To preserve position information, absolute positional encodings [35] are added
to the image and text features. In addition, we add 1D [65] and 2D [17, 78]
relative position bias to image and text features, respectively.

Figure 4: The architecture of the regression-based transformer decoder (a).
The 2D coordinate embedding is obtained by bilinear interpolation from the
nearby grid points, as illustrated in (b).

###  3.4 Regression-based Transformer Decoder

Previous visual seq2seq methods [9, 10, 83, 93, 54] quantize a continuous
coordinate $x$ into a discrete bin $[x]$, introducing an inevitable
quantization error $|x-[x]|$. They formulate coordinate localization as a
classification problem to predict one of $[x]$s, which is suboptimal for
geometric localization. To address this issue, we propose a regression-based
decoder that does not use quantization, and instead predicts the continuous
coordinate values directly (i.e., we use $x$ instead of $[x]$), as shown in
Fig. 4.

#### 2D Coordinate Embedding.

In [9, 10, 83, 93, 54], the coordinate codebook is in 1D space,
$\mathcal{D}\in\mathbb{R}^{B\times C_{e}}$, where $B$ is the number of bins,
$C_{e}$ is the embedding dimension. The embedding of $x$ is obtained by
indexing the codebook, _i.e_., $\mathcal{D}([x])$. To better capture the
geometric relationship between $x$ and $y$ and have a more accurate coordinate
representation, we build a 2D coordinate codebook,
$\mathcal{D}\in\mathbb{R}^{B_{H}\times B_{W}\times C_{e}}$, where $B_{H}$ and
$B_{W}$ are the numbers of bins along height and width dimensions. With this
2D coordinate codebook, we can obtain the precise coordinate embedding for any
floating-point coordinate $(x,y)\in\mathbb{R}^{2}$. First, the floor and
ceiling operations are applied on $(x,y)$ to generate four discrete bins:
$(\underline{x},\underline{y}),(\bar{x},\underline{y}),(\underline{x},\bar{y}),(\bar{x},\bar{y})\in\mathbb{N}^{2}$,
and the corresponding embeddings can be indexed from the 2D codebook, _e.g_.,
$e_{(\underline{x},\underline{y})}=\mathcal{D}(\underline{x},\underline{y})$.
Finally, we get the accurate coordinate embedding $e_{(x,y)}$ by bilinear
interpolation, as in [24]:

$\displaystyle e_{(x,y)}=$ $\displaystyle(\bar{x}-x)(\bar{y}-y)\cdot
e_{(\underline{x},\underline{y})}+(x-\underline{x})(\bar{y}-y)\cdot
e_{(\bar{x},\underline{y})}+$ (2)
$\displaystyle(\bar{x}-x)(y-\underline{y})\cdot
e_{(\underline{x},\bar{y})}+(x-\underline{x})(y-\underline{y})\cdot
e_{(\bar{x},\bar{y})}.$

#### Transformer Decoder Layers.

To capture the relations between multi-modal feature $F_{M}^{N}$ and 2D
coordinate embedding $e_{(x,y)}$, we introduce $N$ transformer decoder layers.
Each transformer layer consists of a multi-head self-attention layer, a multi-
head cross-attention layer and a feed-forward network.

#### Prediction Heads.

Two lightweight heads are built on top of the last decoder layer output
$Q^{N}$ to generate final predictions. The class head is a linear layer that
outputs the token types, indicating whether the current output is a coordinate
token (<COO>), separator token (<SEP>) or an end-of-sequence token (<EOS>):

$\displaystyle\hat{p}$ $\displaystyle=W_{c}Q^{N}+b_{c},$ (3)

where $W_{c}$ and $b_{c}$ are parameters of the linear layer.

The coordinate head is a 3-layer feed-forward network (FFN) with ReLU
activation except for the last layer. It predicts the 2D coordinates of the
referred object bounding box corner points and polygon vertices:

$\displaystyle(\hat{x},\hat{y})$ $\displaystyle=Sigmoid(FFN(Q^{N})).$ (4)

###  3.5 Training

#### Polygon Augmentation.

A polygon is a sparse representation of the dense object contour. Given a
dense contour, the generation of sparse polygons is usually not unique. Given
this property, we introduce a simple yet effective augmentation technique to
increase polygon diversity. As illustrated in Fig. 5, the dense contour is
first interpolated from the original polygon. Then, uniform down-sampling is
applied with an interval randomly sampled from a fixed range to generate
sparse polygons. This creates diverse polygons at different levels of
granularity, and prevents the model from being overfitted to a fixed polygon
representation.

Method | Visual | Text | RefCOCO | RefCOCO+ | RefCOCOg
---|---|---|---|---|---
Backbone | Encoder | val | test A | test B | val | test A | test B | val | test
oIoU | STEP [7] | RN101 | Bi-LSTM | 60.04 | 63.46 | 57.97 | 48.19 | 52.33 | 40.41 | - | -
BRINet [29] | RN101 | LSTM | 60.98 | 62.99 | 59.21 | 48.17 | 52.32 | 42.11 | - | -
CMPC [30] | RN101 | LSTM | 61.36 | 64.53 | 59.64 | 49.56 | 53.44 | 43.23 | - | -
LSCM [31] | RN101 | LSTM | 61.47 | 64.99 | 59.55 | 49.34 | 53.12 | 43.50 | - | -
CMPC+ [49] | RN101 | LSTM | 62.47 | 65.08 | 60.82 | 50.25 | 54.04 | 43.47 | - | -
MCN [57] | DN53 | Bi-GRU | 62.44 | 64.20 | 59.71 | 50.62 | 54.99 | 44.69 | 49.22 | 49.40
EFN [20] | WRN101 | Bi-GRU | 62.76 | 65.69 | 59.67 | 51.50 | 55.24 | 43.01 | - | -
BUSNet [82] | RN101 | Self-Att | 63.27 | 66.41 | 61.39 | 51.76 | 56.87 | 44.13 | - | -
CGAN [56] | DN53 | Bi-GRU | 64.86 | 68.04 | 62.07 | 51.03 | 55.51 | 44.06 | 51.01 | 51.69
LTS [33] | DN53 | Bi-GRU | 65.43 | 67.76 | 63.08 | 54.21 | 58.32 | 48.02 | 54.40 | 54.25
ReSTR [37] | ViT-B | Transformer | 67.22 | 69.30 | 64.45 | 55.78 | 60.44 | 48.27 | - | -
PolyFormer-B | Swin-B | BERT-base | 74.82 | 76.64 | 71.06 | 67.64 | 72.89 | 59.33 | 67.76 | 69.05
PolyFormer-L | Swin-L | BERT-base | 75.96 | 78.29 | 73.25 | 69.33 | 74.56 | 61.87 | 69.20 | 70.19
mIoU | VLT [19] | DN53 | Bi-GRU | 65.65 | 68.29 | 62.73 | 55.50 | 59.20 | 49.36 | 52.99 | 56.65
CRIS [77] | RN101 | GPT-2 | 70.47 | 73.18 | 66.10 | 62.27 | 68.06 | 53.68 | 59.87 | 60.36
SeqTR [93] | DN53 | Bi-GRU | 71.70 | 73.31 | 69.82 | 63.04 | 66.73 | 58.97 | 64.69 | 65.74
RefTr [42] | RN101 | BERT-base | 74.34 | 76.77 | 70.87 | 66.75 | 70.58 | 59.40 | 66.63 | 67.39
LAVT [85] | Swin-B | BERT-base | 74.46 | 76.89 | 70.94 | 65.81 | 70.97 | 59.23 | 63.34 | 63.62
PolyFormer-B | Swin-B | BERT-base | 75.96 | 77.09 | 73.22 | 70.65 | 74.51 | 64.64 | 69.36 | 69.88
PolyFormer-L | Swin-L | BERT-base | 76.94 | 78.49 | 74.83 | 72.15 | 75.71 | 66.73 | 71.15 | 71.17

Table 1: Comparison with the state-of-the-art methods on three referring image
segmentation benchmarks. RN101 denotes ResNet-101 [25], WRN101 refers to Wide
ResNet-101 [89], and DN53 denotes Darknet-53 [66].

#### Objective.

Given an image, a referring expression and preceding tokens, the model is
trained to predict the next token and its type:

$\displaystyle L_{t}$
$\displaystyle=\lambda_{t}L_{coo}((x_{t},y_{t}),(\hat{x}_{t},\hat{y}_{t})|I,T,(x_{i},y_{i})_{i=1:t-1})$
(5)
$\displaystyle\cdot\mathbb{I}[p_{t}==\texttt{<COO>}]+\lambda_{cls}L_{cls}(p_{t},\hat{p}_{t}|I,T,p_{1:t-1}),$

where

$\lambda_{t}=\left\\{\begin{aligned} \lambda_{box},&~{}~{}~{}~{}t\leq 2,\\\
\lambda_{poly},&~{}~{}~{}~{}otherwise,\\\ \end{aligned}\right.$

$L_{coo}$ is the $L_{1}$ regression loss, $L_{cls}$ is the label smoothed
cross-entropy loss, and $\mathbb{I}[\cdot]$ is the indicator function. The
regression loss is only computed for the coordinate tokens, where
$\lambda_{box}$ and $\lambda_{poly}$ are the corresponding token weights. The
total loss is the sum of $L_{t}$ over all tokens in a sequence.

#### Inference.

During inference, we start the generation by inputting the <BOS> token. First,
we get the token type from the class head. If it is a coordinate token, we
will obtain the 2D coordinate prediction from the coordinate head conditioned
on the preceding predictions; if it is a separator token, it indicates the end
of the preceding polygon, so the separator token will be added to the output
sequence. This sequential prediction will stop once <EOS> is output. In the
generated sequence, the first two tokens are bounding box coordinates and the
rest are polygon vertices. Final segmentation mask is obtained from the
polygon predictions.

Figure 5:  Illustration of polygon augmentation. Polygons at different levels
of granularity (c) - (d) are sampled from dense contour (b) that is
interpolated from the original polygon (a).

##  4 Experimental Results

###  4.1 Datasets and Metrics

#### Datasets.

The experiments are conducted on three major benchmarks for RIS and REC:
RefCOCO [88], RefCOCO+ [88], and RefCOCOg [58, 60]. RefCOCO has 142,209
annotated expressions for 50,000 objects in 19,994 images, and RefCOCO+
consists of 141,564 expressions for 49,856 objects in 19,992 images. Compared
with RefCOCO, location words are absent from the referring expressions in
RefCOCO+, which makes it more challenging. RefCOCOg consists of 85,474
referring expressions for 54,822 objects in 26,711 images. It contains
referring expressions collected on Amazon Mechanical Turk, and therefore the
descriptions are longer and more complex (8.4 words on average compared with
3.5 words of RefCOCO and RefCOCO+). We use the UMD split for RefCOCOg [60].

#### Evaluation Metrics.

We use mean Intersection-over-Union (mIoU) as the evaluation metric for RIS.
For a fair comparison, we also use overall Intersection-over-Union (oIoU) when
comparing with papers that only report oIoU results. Additionally, we also
evaluate PolyFormer on the REC task as it is a unified framework for both RIS
and REC tasks. We adopt the standard metric<EMAIL_ADDRESS>[93, 76], where a
prediction is considered correct if its Intersection-over-Union (IoU) with
ground-truth box is higher than 0.5.

Method | Visual | Text | RefCOCO | RefCOCO+ | RefCOCOg
---|---|---|---|---|---
Backbone | Encoder | val | test A | test B | val | test A | test B | val | test
UNTIER-L [11] | RN101 | BERT | 81.41 | 87.04 | 74.17 | 75.90 | 81.45 | 66.70 | 74.86 | 75.77
VILLA-L [21] | RN101 | BERT | 82.39 | 87.48 | 74.84 | 76.17 | 81.54 | 66.84 | 76.18 | 76.71
RefTr [42] | RN101 | BERT-base | 85.65 | 88.73 | 81.16 | 77.55 | 82.26 | 68.99 | 79.25 | 80.01
SeqTR [93] | DN53 | Bi-GRU | 87.00 | 90.15 | 83.59 | 78.69 | 84.51 | 71.87 | 82.69 | 83.37
MDETR [34] | ENB3 | RoBERTa-base | 87.51 | 90.40 | 82.67 | 81.13 | 85.52 | 72.96 | 83.35 | 83.31
OFA-B [76] | RN101 | Embedding layer | 88.48 | 90.67 | 83.30 | 81.39 | 87.15 | 74.29 | 82.29 | 82.31
UniTAB [83] | RN101 | RoBERT-base | 88.59 | 91.06 | 83.75 | 80.97 | 85.36 | 71.55 | 84.58 | 84.70
OFA-L [76] | RN152 | Embedding layer | 90.05 | 92.93 | 85.26 | 85.80 | 89.87 | 79.22 | 85.89 | 86.55
PolyFormer-B | Swin-B | BERT-base | 89.73 | 91.73 | 86.03 | 83.73 | 88.60 | 76.38 | 84.46 | 84.96
PolyFormer-L | Swin-L | BERT-base | 90.38 | 92.89 | 87.16 | 84.98 | 89.77 | 77.97 | 85.83 | 85.91

Table 2: Comparison with the state-of-the-art methods on three referring
expression comprehension benchmarks. ENB3 denotes EfficientNet-B3 [75].

###  4.2 Implementation Details

Model Settings. In PolyFormer-B, we use Swin-B [51] as the visual encoder and
BERT-base [18] as the text encoder. For the transformer encoder and decoder,
we adopt 6 encoder layers and 6 decoder layers. To investigate the impact of
different model scales, we develop a larger model, PolyFormer-L, that adopts
Swin-L as the visual backbone with 12 transformer encoder and decoder layers.

#### Training Details.

To leverage large-scale image-text dataset with grounded box annotations, we
first pre-train PolyFormer on the REC task with the combination of Visual
Genome [38], RefCOCO [88], RefCOCO+ [88], RefCOCOg [58, 60], and
Flickr30k-entities [62]. During the multi-task fine-tuning stage, we train the
model for both RIS and REC on a combined training dataset [4] of RefCOCO,
RefCOCO+, and RefCOCOg with all validation and testing images removed. We use
the AdamW optimizer [52] with $(\beta_{1},\beta_{2})=(0.9,0.999)$ and
$\epsilon=1\times 10^{-8}$. The initial learning rate is $5\times 10^{-5}$
with polynomial learning rate decay. We train the model for 20 epochs during
pre-training and 100 epochs for fine-tuning with a batch size of 160 and 128,
respectively. The coefficients for losses are set as $\lambda_{box}=0.1$,
$\lambda_{poly}=1$ and $\lambda_{cls}=5\times 10^{-4}$. Images are resized to
$512\times 512$, and polygon augmentation is applied with a probability of
50%. The 2D coordinate embedding codebook is constructed with $64\times 64$
bins.

###  4.3 Main Results

#### Referring Image Segmentation.

We compare PolyFormer with the state-of-the-art methods in Table 1. It can be
observed that PolyFormer models outperform previous methods on each split of
the three datasets under all metrics by a clear margin. First, on the RefCOCO
dataset, compared with the recent LAVT [85], PolyFormer-B achieves better
results with absolute mIoU gains of 1.5%, 0.2%, and 2.28% on the three splits.
Second, on the more challenging RefCOCO+ dataset, PolyFormer-B significantly
outperforms the previous state-of-the-art RefTr [42] by absolute mIoU margins
of 3.9%, 3.93%, 5.24% on the validation, test A and test B sets, respectively.
Third, on the most challenging RefCOCOg dataset where language expressions are
longer and more complex, PolyFormer-B achieves notable performance
improvements of 2.73% and 2.49% mIoU points on the validation and test sets
compared with the second-best method RefTr [42]. Using the stronger Swin-L
backbone and a larger encoder and decoder, PolyFormer-L achieves consistent
performance gains of around 1$\sim$2 absolute points over PolyFormer-B across
all datasets. These results demonstrate the superiority of our polygon-based
method PolyFormer over the previous mask-based methods.

Method | Visual Backbone | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$
---|---|---|---|---
CMSA+RNN [86] | ResNet-50 | 40.2 | 36.9 | 43.5
URVOS [71] | ResNet-50 | 51.5 | 47.3 | 56.0
CITD [44] | ResNet-101 | 56.4 | 54.8 | 58.1
ReferFormer [79] | Swin-L | 60.5 | 57.6 | 63.4
ReferFormer [79] | Video-Swin-B | 61.1 | 58.1 | 64.1
PolyFormer-B† | Swin-B | 60.9 | 56.6 | 65.2
PolyFormer-L† | Swin-L | 61.5 | 57.2 | 65.8

Table 3: Comparison with the state-of-the-art methods on Ref-DAVIS17. †means
our model is trained on image datasets only. ReferFormer is trained on both
image and video datasets.

#### Referring Expression Comprehension.

We further evaluate PolyFormer on the REC datasets and compare the performance
with other state-of-the-art methods in Table 2. OFA [76] is a foundation model
which provides a unified interface for a wide range of tasks from different
modalities. Compared with OFA-L, PolyFormer-L achieves better results on
RefCOCO and comparable results on the remaining two datasets. Note that OFA
utilizes 19M image-text data for pre-training, while PolyFormer only uses 6M
image-text pairs. Another recent paper SeqTR [93] also proposes a seq2seq
model for multi-task learning of RIS and REC, but they observe performance
degradation for each task when trained jointly. PolyFormer-B significantly
outperforms SeqTR by absolute margins of 2.73%, 5.04% and 1.77% on the three
validation sets. These results illustrate the effectiveness of our unified
framework.

#### Zero-shot Transfer to Referring Video Object Segmentation.

To further test its generalization ability, we evaluate PolyFormer on the
referring video object segmentation dataset Ref-DAVIS17 [36] in a zero-shot
manner. We simply view the video data as a sequence of images, and apply
PolyFormer on the video frame-by-frame. As shown in Table 3, PolyFormer-L
achieves 61.5% $\mathcal{J}\&\mathcal{F}$ without any finetuning on the video
data. It is even better than the state-of-the-art ReferFormer [79], which is
fully trained on referring video segmentation data.

###  4.4 Ablation Studies

In this section, we perform extensive ablation studies on the RefCOCO,
RefCOCO+ and RefCOCOg validation sets to study the effects of core components
of PolyFormer. All ablation experiments are performed on PolyFormer-B.

Decoder | RefCOCO | RefCOCO+ | RefCOCOg
---|---|---|---
RIS | Classification | 74.11 | 68.79 | 67.69
Regression |  75.96 (+1.85) |  70.65 (+1.86) |  69.36 (+1.67)
REC | Classification | 87.03 | 81.35 | 82.21
Regression |  89.73 (+2.70) |  83.73 (+2.38) |  84.46 (+2.25)

Table 4:  Ablation study on regression-based decoder.

#### Coordinate Classification vs. Regression.

We implement a classification-based model for coordinate prediction following
[9, 83, 93]. Specifically, the coordinates are quantized into discrete bins
and a classifier is adopted to output the discrete tokens. As shown in Table
4, the regression-based model consistently outperforms the classification-
based model on all datasets, e.g., +1.86 for RIS and +2.38 for REC on the
RefCOCO+ dataset. This shows that the regression-based model is a better
choice for geometric localization tasks than the classification-based
counterpart.

Order | Aug | Multi-task | <SEP> | RefCOCO | RefCOCO+ | RefCOCOg
---|---|---|---|---|---|---
✗ | ✗ | ✗ | ✗ | 55.92 | 51.64 | 50.65
✓ | ✗ | ✗ | ✗ | 68.35 | 63.28 | 62.18
✓ | ✓ | ✗ | ✗ | 72.07 | 66.68 | 65.20
✓ | ✓ | ✓ | ✗ | 75.14 | 69.86 | 68.70
✓ | ✓ | ✓ | ✓ | 75.96 | 70.65 | 69.36

Table 5:  Ablation study on target sequence construction.

Expression: “an Asian girl with a pink shirt eating at the table”
---
|  |  |  |  |  |

(a) $t_{poly}=1$ (Start)

(b) $t_{poly}=5$

(c) $t_{poly}=9$

(d) $t_{poly}=13$

(e) $t_{poly}=17$

(f) $t_{poly}=20$

(g) $t_{poly}=23$ (End)

Figure 6: The cross-attention maps of the decoder when generating the polygon.
$\star$ is the 2D vertex prediction at inference step $t_{poly}$.

LAVT |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
SeqTR |  |  |  |  |  |  |
PolyFormer |  |  |  |  |  |  |

(a) (a) “a dark grey dog on a light grey round bed wearing a red collar”

(b) (b) “girl in purple”

(c) (c) “zebra eating grass with a goose in front of it”

(d) (d) “a black car parked at a transportation terminal”

(e) (e) “A gentleman otter in a 19th century portrait”

(f) (f) “A pikachu fine-dining with a view to the Eiffel Tower”

(g) (g) “A pig robot preparing a delicious meal”

Figure 7:  The results of LAVT [85] (top), SeqTR [93] (middle), and PolyFormer
(bottom) on RefCOCOg test set (a-d) and images generated by [68] (e-g). LAVT
is for referring image segmentation only. For SeqTR, we generate the bounding
boxes and segmentation masks from the task-specific models as they perform
better than the multi-task variant.

#### Component Analysis of Target Sequence Construction.

We study the effects of several components in constructing the target
sequence, including (1) polygon ordering, where the polygon vertices are
ordered clock-wise and the starting point is set as the vertex closest to the
image origin; (2) data augmentation, where we generate polygons at different
levels of granularity on-the-fly during training; (3) multi-task learning of
RIS and REC; and (4) separator token, where we add <SEP> token to handle
multi-polygon cases. The results are shown in Table 5. Using randomly ordered
polygons only achieve 55.92% mIoU on RefCOCO. Polygon ordering is essential
and leads to a substantial improvement of 12.43%. Both polygon augmentation
and multi-task learning are beneficial, with gains of 3.72% and 3.07%,
respectively. The separator token <SEP> brings 0.82% increase of mIoU. This
small gain is reasonable considering only a small number of samples have
multiple polygons. We observe similar trends in the remaining two datasets.

###  4.5 Visualization Results

#### Cross-attention Map.

When generating the vertex tokens, the regression-based decoder computes the
self-attention over the preceding tokens and cross-attention over the multi-
modal feature. Here we visualize the cross-attention map (averaged over all
layers and heads) when the model predicts a new token. Fig. 6 shows the cross-
attention maps at different steps of the polygon generation. We observe that
the cross-attention map concentrates on the object referred by the sentence,
and moves around the object boundary during the polygon generation process.

#### Prediction Visualization.

We show the visualization results of PolyFormer on RefCOCOg test set in Fig. 7
(a)-(d). It can be seen that PolyFormer is able to segment the referred object
in challenging scenarios, e.g., instances with occlusion and complex shapes,
instances that are partially displayed or require complex language
understanding. We also show the results on images generated by Stable
Diffusion [68] in Fig. 7 (e)-(g). PolyFormer demonstrates good generalization
ability on synthetic images and text descriptions that have never been seen
during training. In contrast, the state-of-the-arts LAVT [85] and SeqTR [93]
fail to generate satisfactory results. More visualization results are provided
in the supplementary material.

##  5 Conclusion

In this work, we propose PolyFormer, a simple and unified framework for
referring image segmentation and referring expression comprehension. It is a
sequence to sequence framework which can naturally fuse multi-modal features
as the input sequence and multi-task predictions as the output sequence.
Moreover, we design a novel regression-based decoder to generate continuous 2D
coordinates without quantization errors. PolyFormer achieves competitive
results for RIS and REC and shows good generalization to unseen scenarios. We
believe this simple framework can be extended to other tasks beyond RIS and
REC.

#### Limitations and Broader Impacts.

The training of PolyFormer requires accurate bounding box and polygon
annotations. How to reduce such dependence and utilize weakly-supervised data
for region-level image understanding needs further exploration. For the data
and model, we need to further understand the broader impacts including but not
limited to fairness, social bias and potential misuse.

## References

  * [1] David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler.  Efficient interactive annotation of segmentation datasets with Polygon-RNN++.  In CVPR, pages 859–868, 2018.
  * [2] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  YOLACT: Real-time instance segmentation.  In ICCV, pages 9157–9166, 2019.
  * [3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [4] Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rahul Bhotika, and Stefano Soatto.  X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks.  In ECCV, pages 290–308, 2022.
  * [5] Zhaowei Cai and Nuno Vasconcelos.  Cascade R-CNN: Delving into high quality object detection.  In CVPR, pages 6154–6162, 2018.
  * [6] Lluis Castrejon, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler.  Annotating object instances with a Polygon-RNN.  In CVPR, pages 5230–5238, 2017.
  * [7] Ding-Jie Chen, Songhao Jia, Yi-Chen Lo, Hwann-Tzong Chen, and Tyng-Luh Liu.  See-through-text grouping for referring image segmentation.  In ICCV, pages 7454–7463, 2019.
  * [8] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al.  Hybrid task cascade for instance segmentation.  In CVPR, pages 4974–4983, 2019.
  * [9] Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, and Geoffrey Hinton.  Pix2seq: A language modeling framework for object detection.  In ICLR, 2022.
  * [10] Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J. Fleet, and Geoffrey Hinton.  A unified sequence interface for vision tasks.  In NeurIPS, 2022.
  * [11] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  UNITER: UNiversal Image-TExt Representation Learning.  In ECCV, pages 104–120, 2020.
  * [12] Yi-Wen Chen, Yi-Hsuan Tsai, Tiantian Wang, Yen-Yu Lin, and Ming-Hsuan Yang.  Referring expression object segmentation with caption-aware consistency.  In BMVC, 2019.
  * [13] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio.  On the properties of neural machine translation: Encoder-decoder approaches.  In SSST@EMNLP, pages 103–111, 2014.
  * [14] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In EMNLP, pages 1724–1734, 2014.
  * [15] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, pages 3213–3223, June 2016.
  * [16] Jifeng Dai, Kaiming He, and Jian Sun.  Instance-aware semantic segmentation via multi-task network cascades.  In CVPR, pages 3150–3158, 2016.
  * [17] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan.  Coatnet: Marrying convolution and attention for all data sizes.  In NeurIPS, 2021.
  * [18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In NAACL-HLT, pages 4171–4186, 2019.
  * [19] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang.  Vision-language transformer and query generation for referring segmentation.  In ICCV, pages 16321–16330, 2021.
  * [20] Guang Feng, Zhiwei Hu, Lihe Zhang, and Huchuan Lu.  Encoder fusion network with co-attention embedding for referring image segmentation.  In CVPR, pages 15506–15515, 2021.
  * [21] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu.  Large-scale adversarial training for vision-and-language representation learning.  In NeurIPS, 2020.
  * [22] Spyros Gidaris and Nikos Komodakis.  LocNet: Improving localization accuracy for object detection.  In CVPR, pages 789–798, 2016.
  * [23] Ross Girshick.  Fast R-CNN.  In ICCV, pages 1440–1448, 2015.
  * [24] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In ICCV, pages 2961–2969, 2017.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [26] Richang Hong, Daqing Liu, Xiaoyu Mo, Xiangnan He, and Hanwang Zhang.  Learning to compose and reason with language tree structures for visual grounding.  PAMI, 44(2):684–696, 2019.
  * [27] Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko.  Modeling relationships in referential expressions with compositional modular networks.  In CVPR, pages 1115–1124, 2017.
  * [28] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell.  Segmentation from natural language expressions.  In ECCV, pages 108–124, 2016.
  * [29] Zhiwei Hu, Guang Feng, Jiayu Sun, Lihe Zhang, and Huchuan Lu.  Bi-directional relationship inferring network for referring image segmentation.  In CVPR, pages 4424–4433, 2020.
  * [30] Shaofei Huang, Tianrui Hui, Si Liu, Guanbin Li, Yunchao Wei, Jizhong Han, Luoqi Liu, and Bo Li.  Referring image segmentation via cross-modal progressive comprehension.  In CVPR, pages 10488–10497, 2020.
  * [31] Tianrui Hui, Si Liu, Shaofei Huang, Guanbin Li, Sansi Yu, Faxi Zhang, and Jizhong Han.  Linguistic structure guided context modeling for referring image segmentation.  In ECCV, pages 59–75, 2020.
  * [32] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial transformer networks.  In NeurIPS, 2015.
  * [33] Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, and Tieniu Tan.  Locate then segment: A strong pipeline for referring image segmentation.  In CVPR, pages 9858–9867, 2021.
  * [34] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  MDETR - modulated detection for end-to-end multi-modal understanding.  In ICCV, pages 1780–1790, 2021.
  * [35] Guolin Ke, Di He, and Tie-Yan Liu.  Rethinking positional encoding in language pre-training.  In ICLR, 2021.
  * [36] Anna Khoreva, Anna Rohrbach, and Bernt Schiele.  Video object segmentation with language referring expressions.  In ACCV, pages 123–141, 2018.
  * [37] Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak.  Restr: Convolution-free referring image segmentation using transformers.  In CVPR, pages 18145–18154, 2022.
  * [38] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  IJCV, 123(1):32–73, 2017.
  * [39] Justin Lazarow, Weijian Xu, and Zhuowen Tu.  Instance segmentation with mask-supervised polygonal boundary transformers.  In CVPR, pages 4372–4381, 2022.
  * [40] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In NeurIPS, 2021.
  * [41] Mengtian Li, Laszlo Jeni, and Deva Ramanan.  Brute-force facial landmark analysis with a 140,000-way classifier.  In AAAI, volume 32, 2018.
  * [42] Muchen Li and Leonid Sigal.  Referring transformer: A one-step approach to multi-task visual grounding.  In NeurIPS, 2021.
  * [43] Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia.  Referring image segmentation via recurrent refinement networks.  In CVPR, pages 5745–5753, 2018.
  * [44] Chen Liang, Yu Wu, Tianfei Zhou, Wenguan Wang, Zongxin Yang, Yunchao Wei, and Yi Yang.  Rethinking cross-modal interaction from a top-down perspective for referring video object segmentation.  arXiv preprint arXiv:2106.01061, 2021.
  * [45] Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, and Raquel Urtasun.  PolyTransform: Deep Polygon Transformer for Instance Segmentation.  In CVPR, pages 9128–9137, 2020.
  * [46] Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo Li.  A real-time cross-modality correlation filtering method for referring expression comprehension.  In CVPR, pages 10880–10889, 2020.
  * [47] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common Objects in Context.  In ECCV, pages 740–755, 2014.
  * [48] Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan Yuille.  Recurrent multimodal interaction for referring image segmentation.  In ICCV, pages 1271–1280, 2017.
  * [49] Si Liu, Tianrui Hui, Shaofei Huang, Yunchao Wei, Bo Li, and Guanbin Li.  Cross-modal progressive comprehension for referring segmentation.  PAMI, 44(9):4761–4775, 2022.
  * [50] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia.  Path aggregation network for instance segmentation.  In CVPR, pages 8759–8768, 2018.
  * [51] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, pages 10012–10022, 2021.
  * [52] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [53] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In NeurIPS, 2019.
  * [54] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Unified-IO: A unified model for vision, language, and multi-modal tasks.  arXiv preprint arXiv:2206.08916, 2022.
  * [55] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-task vision and language representation learning.  In CVPR, pages 10437–10446, 2020.
  * [56] Gen Luo, Yiyi Zhou, Rongrong Ji, Xiaoshuai Sun, Jinsong Su, Chia-Wen Lin, and Qi Tian.  Cascade grouped attention network for referring expression segmentation.  In ACMMM, pages 1274–1282, 2020.
  * [57] Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji.  Multi-task collaborative network for joint referring expression comprehension and segmentation.  In CVPR, pages 10034–10043, 2020.
  * [58] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy.  Generation and comprehension of unambiguous object descriptions.  In CVPR, pages 11–20, 2016.
  * [59] Edgar Margffoy-Tuay, Juan C Pérez, Emilio Botero, and Pablo Arbeláez.  Dynamic multimodal instance segmentation guided by natural language queries.  In ECCV, pages 630–645, 2018.
  * [60] Varun K Nagaraja, Vlad I Morariu, and Larry S Davis.  Modeling context between objects for referring expression understanding.  In ECCV, pages 792–807, 2016.
  * [61] Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, and Xiaowei Zhou.  Deep snake for real-time instance segmentation.  In CVPR, pages 8530–8539, 2020.
  * [62] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In ICCV, pages 2641–2649, 2015.
  * [63] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 DAVIS challenge on video object segmentation.  arXiv preprint arXiv:1704.00675, 2017.
  * [64] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [65] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  JMLR, 21(140):1–67, 2020.
  * [66] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [67] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster R-CNN: Towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [68] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, pages 10684–10695, 2022.
  * [69] Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman.  LabelMe: a database and web-based tool for image annotation.  IJCV, 77(1):157–173, 2008.
  * [70] Jürgen Schmidhuber, Sepp Hochreiter, et al.  Long short-term memory.  Neural Comput, 9(8):1735–1780, 1997.
  * [71] Seonguk Seo, Joon-Young Lee, and Bohyung Han.  Urvos: Unified referring video object segmentation network with a large-scale benchmark.  In ECCV, pages 208–223, 2020.
  * [72] Hengcan Shi, Hongliang Li, Fanman Meng, and Qingbo Wu.  Key-word-aware network for referring expression image segmentation.  In ECCV, pages 38–54, 2018.
  * [73] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  Sequence to sequence learning with neural networks.  In NeurIPS, 2014.
  * [74] Hao Tan and Mohit Bansal.  LXMERT: Learning cross-modality encoder representations from transformers.  In EMNLP-IJCNLP, pages 5100–5111, 2019.
  * [75] Mingxing Tan and Quoc Le.  EfficientNet: Rethinking model scaling for convolutional neural networks.  In ICML, pages 6105–6114, 2019.
  * [76] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In ICML, pages 23318–23340, 2022.
  * [77] Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu.  CRIS: Clip-driven referring image segmentation.  In CVPR, pages 11686–11695, 2022.
  * [78] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  SimVLM: Simple visual language model pretraining with weak supervision.  In ICLR, 2022.
  * [79] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo.  Language as queries for referring video object segmentation.  In CVPR, pages 4974–4984, 2022.
  * [80] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo.  Polarmask: Single shot instance segmentation with polar representation.  In CVPR, pages 12193–12202, 2020.
  * [81] Chenyang Xu and J.L. Prince.  Snakes, shapes, and gradient vector flow.  TIP, 7(3):359–369, 1998.
  * [82] Sibei Yang, Meng Xia, Guanbin Li, Hong-Yu Zhou, and Yizhou Yu.  Bottom-up shift and reasoning for referring image segmentation.  In CVPR, pages 11266–11275, 2021.
  * [83] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Unitab: Unifying text and box outputs for grounded vision-language modeling.  In ECCV, pages 521–539, 2022.
  * [84] Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo.  A fast and accurate one-stage approach to visual grounding.  In ICCV, pages 4683–4693, 2019.
  * [85] Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr.  LAVT: Language-Aware Vision Transformer for Referring Image Segmentation.  In CVPR, pages 18155–18165, 2022.
  * [86] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang.  Cross-modal self-attention network for referring image segmentation.  In CVPR, pages 10502–10511, 2019.
  * [87] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg.  MAttNet: Modular attention network for referring expression comprehension.  In CVPR, pages 1307–1315, 2018.
  * [88] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg.  Modeling context in referring expressions.  In ECCV, pages 69–85, 2016.
  * [89] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In BMVC, pages 87.1–87.12, 2016.
  * [90] Hanwang Zhang, Yulei Niu, and Shih-Fu Chang.  Grounding referring expressions in images by variational context.  In CVPR, pages 4158–4166, 2018.
  * [91] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In CVPR, pages 5579–5588, 2021.
  * [92] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.
  * [93] Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji.  SeqTR: A Simple Yet Universal Network for Visual Grounding.  In ECCV, pages 598–615, 2022.
  * [94] Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and Anton Van Den Hengel.  Parallel attention: A unified framework for visual object discovery through dialogs and queries.  In CVPR, pages 4252–4261, 2018.

## Appendix

##  Appendix A Additional Dataset Details

We evaluate PolyFormer on three benchmark image datasets, RefCOCO [88],
RefCOCO+ [88], RefCOCOg [58, 60]. All images of RefCOCO, RefCOCO+, and
RefCOCOg are from the MS COCO dataset [47] and annotated with referring
expressions. We further evaluate PolyFormer models for the Referring Video
Object Segmentation (R-VOS) task on Ref-DAVIS17 [36].

####  RefCOCO/RefCOCO+:

These two datasets are collected using a two-player game[88]. RefCOCO has
142,209 annotated expressions for 50,000 objects in 19,994 images, and
RefCOCO+ consists of 141,564 expressions for 49,856 objects in 19,992 images.
These two datasets are splitted into training, validation, test A and test B
sets, where test A contains images of multiple people and test B contains
images of multiple instances of all other objects. Compared to RefCOCO,
location words are banned from the referring expressions in RefCOCO+, which
makes it more challenging.

####  RefCOCOg:

This dataset is collected on Amazon Mechanical Turk, where workers are asked
to write natural language referring expressions for objects. RefCOCOg consists
of 85,474 referring expressions for 54,822 objects in 26,711 images. RefCOCOg
has longer, more complex expressions (8.4 words on average), while the
expressions in RefCOCO and RefCOCO+ are more succinct (3.5 words on average),
which makes RefCOCOg particularly challenging. We use the UMD partition [60]
for RefCOCOg as it provides both validation and testing sets and there is no
overlapping between training and validation images.

####  Ref-DAVIS17:

Ref-DAVIS17 contains 90 videos from the DAVIS17 [63] dataset, where language
descriptions are provided for specific objects in each video. It contains
1,544 referring expressions for 205 objects. The dataset is split into a
training set and a validation set, containing 60 and 30 videos respectively.
For each referred object, each of the two annotators provides the descriptions
of the first-frame and the full-video. For the Ref-DAVIS17 dataset, we use the
standard evaluation metrics: Region Jaccard ($\mathcal{J}$), Boundary F
measure ($\mathcal{F}$), and their average value ($\mathcal{J}\&\mathcal{F}$).

##  Appendix B Additional Implementation Details

The dimension of image feature $C_{v}$ is 1024 for PolyFormer-B and 1536 for
PolyFormer-L. The dimensions of language feature $C_{l}$ and coordinate
embedding $C_{e}$ are 768. We use a linear layer to project the language and
image features into the same dimension of 768. We adopt 12 attention heads in
the self-attention and cross-attention layers, and GELU activations in the
transformer encoder and decoder layers. For $L_{cls}$, we set the label
smoothing factor to 0.1.

##  Appendix C Additional Experiment Results

To obtain the accurate coordinate embedding, we build a 2D coordinate
codebook, $\mathcal{D}\in\mathbb{R}^{B_{H}\times B_{W}\times C_{e}}$, where
$B_{H}$ and $B_{W}$ are the numbers of bins along the height and width
dimensions, respectively. We train PolyFormer-B models with different number
of bins $B_{H}\times B_{W}$ and the results are summarized in Table 6. We
observe that using coordinate book with $64\times 64$ bins achieves the best
result, which is adopted by default in all the other experiments.

$B_{H}\times B_{W}$ | RefCOCO | RefCOCO+ | RefCOCOg
---|---|---|---
$32\times 32$ | 75.07 | 70.15 | 68.49
$64\times 64$ | 75.96 | 70.65 | 69.36
$128\times 128$ | 74.99 | 70.01 | 68.69

Table 6: Ablation study on the size of 2D coordinate codebook.

Expression: “a without hairy brown color teddy bear”
---
|  |  |  |  |  |
Expression: “a chili dog with slices of cheese visible under the chili”
|  |  |  |  |  |
Expression: “the orange closest to the banana”
|  |  |  |  |  |

(a) $t_{poly}=1$ (start)

(b) $t_{poly}=6$

(c) $t_{poly}=9$

(d) $t_{poly}=13$

(e) $t_{poly}=17$

(f) $t_{poly}=21$

(g) $t_{poly}=24$ (end)

(h) $t_{poly}=1$ (start)

(i) $t_{poly}=6$

(j) $t_{poly}=11$

(k) $t_{poly}=16$

(l) $t_{poly}=21$

(m) $t_{poly}=26$

(n) $t_{poly}=34$ (end)

(o) $t_{poly}=1$ (start)

(p) $t_{poly}=4$

(q) $t_{poly}=7$

(r) $t_{poly}=10$

(s) $t_{poly}=13$

(t) $t_{poly}=16$

(u) $t_{poly}=19$ (end)

Figure 8: Decoder’s cross-attention map when predicting the polygons. $\star$
indicates the vertex prediction at time step $t_{poly}$.

LAVT [85] |  |  |  |  |  |
---|---|---|---|---|---|---
SeqTR [93] |  |  |  |  |  |
PolyFormer |  |  |  |  |  |

(a) (a) “A cat chef cooking fish in a fancy restaurant”

(b) (b) “A chair that looks like octopus”

(c) (c) “A small cabin on top of a snowy mountain in the style of Disney
artstation”

(d) (d) “A shiba inu puppy painted by Monet”

(e) “A unicorn doing computer vision research”

(f) “A bear astronaut in the space”

Figure 9: The result comparison of LAVT [85], SeqTR [93] and PolyFormer on
synthetic images generated by Stable Diffusion [68].

LAVT [85] |  |  |  |  |  |
---|---|---|---|---|---|---
SeqTR [93] |  |  |  |  |  |
PolyFormer |  |  |  |  |  |
LAVT [85] |  |  |  |  |  |
SeqTR [93] |  |  |  |  |  |
PolyFormer |  |  |  |  |  |

(a) “horse on the left of the group of horses”

(b) “small green vase on the left with a flower in it”

(c) “the elephant with the baby elephant”

(d) “the taller giraffe”

(e) “a white baseball bat, held by a person”

(f) “a red and black motorcycle with a Santa riding it”

(g) “old yellow and white truck parked behind other truck”

(h) “boy with blue plaid shirt and glasses”

(i) “the surfboard the woman in a white shirt and blue capris is holding”

(j) (e) “a zebra with its head not visible but much of its body able to be
seen”

(k) (f) “ a girl was cooking the food and serving”

(l) (g) “a man wearing a black shirt and a black and white striped apron
stirring something in a metal container”

Figure 10: The result comparison of LAVT [85], SeqTR [93] and PolyFormer on
RefCOCOg test set. PolyFormer simultaneously predicts the bounding box and
polygon vertices that forms the segmentation mask. LAVT is for referring image
segmentation only. For SeqTR, we generate the bounding boxes and segmentation
masks from the task-specific models as they perform better than the multi-task
model.

##  Appendix D More Visualization Results

###  D.1 Cross-attention Map

More cross-attention map visualization is shown in Fig. 8. We observe that the
cross-attention map concentrates on the object referred by the sentence, and
moves around the object boundary during the polygon generation process.

###  D.2 Prediction Visualization

Fig. 9 shows more examples on the synthetic images generated by Stable
Diffusion [68]. Fig. 10 shows more examples on the RefCOCOg test set. It can
be seen that PolyFormer is able to segment the referred object in challenging
scenarios, e.g., instances with occlusion and complex shapes, instances that
are partially displayed or require complex language understanding. In
addition, PolyFormer demonstrates good generalization ability on synthetic
images and text descriptions that have never been seen during training. In
contrast, the state-of-the-arts LAVT [85] and SeqTR [93] fail to generate
satisfactory results.


# Performance evaluation of deep segmentation models on Landsat-8 imagery

Akshat Bhandari1
Dept. of Computer Science and Engineering
<EMAIL_ADDRESS>
&Sriya Rallabandi11footnotemark: 1122footnotemark: 2
Dept. of Information and Communication Technology
<EMAIL_ADDRESS>
&Sanchit Singhal11footnotemark: 1122footnotemark: 2
Dept. of Electrical and Electronics Engineering
<EMAIL_ADDRESS>
&Aditya Kasliwal11footnotemark: 1122footnotemark: 2
Dept. of Data Science and Computer Applications
<EMAIL_ADDRESS>
&Pratinav Seth11footnotemark: 1
Dept. of Data Science and Computer Applications
<EMAIL_ADDRESS>
Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal,
IndiaAuthors have contributed equally to this work

###### Abstract

Contrails, short for condensation trails, are line-shaped ice clouds produced
by aircraft engine exhaust when they fly through cold and humid air. They
generate a greenhouse effect by absorbing or directing back to Earth
approximately 33% of emitted outgoing longwave radiation[1]. They account for
over half of the climate change resulting from aviation activities. Avoiding
contrails and adjusting flight routes could be an inexpensive and effective
way to reduce their impact. An accurate, automated, and reliable detection
algorithm is required to develop and evaluate contrail avoidance strategies.
Advancement in contrail detection has been severely limited due to several
factors, primarily due to a lack of quality-labeled data. Recently, [2]
proposed a large human-labeled Landsat-8 contrails dataset. Each contrail is
carefully labeled with various inputs in various scenes of Landsat-8 satellite
imagery. In this work, we benchmark several popular segmentation models with
combinations of different loss functions and encoder backbones. This work is
the first to apply state-of-the-art segmentation techniques to detect
contrails in low-orbit satellite imagery. Our work can also be used as an open
benchmark for contrail segmentation and is available at
https://github.com/Kasliwal17/Contrail_Segmentation.

##  1 Introduction

Contrails, or vapor trails, usually form when water vapor from the exhaust
combines with the low ambient temperatures in high-altitude regions. These
contrails create an additional blanket of clouds, increasing the heat-trapping
effect that directly contributes to global warming. Although some may be
short-lived, they can, in many cases, linger for hours[3], which results in
new contrails created by aircraft flying on the same routes, accumulating and
forming a ’contrail cirrus’[4][5], which worsens the heat-trapping effect. As
a result, contrails have become a new cause of alarm for climate change as
they have become one of the most significant contributors to global warming
caused by the aviation industry[6].

An effective solution that can reduce the effect of contrails is to avoid the
regions of the atmosphere that are cold and humid enough to create
contrails[7] and to adjust flight routes and slight changes in flight
altitudes[8] to reduce the air traffic in such regions. This can be a
relatively easy solution to prevent contrail formations and their effects,
given that a comparably small proportion of aircraft contributes majorly.
Physics-based models such as CoCIP[9] and APCEMM[10] simulate contrails’
microphysics and radiative transfer effects but suffer from uncertainties in
their inputs. These uncertainties -such as the spatial resolution of numerical
weather models and limitations of high-altitude humidity measurements[11]) -
result in large errors in contrail RF predictions from such models. Contrail
detection work in aerial images can be largely attributed to Mannstein et
al.[12][13][14][15][16][17][18]. This algorithm operates on brightness
temperature imagery using a sequence of manually designed convolution and
thresholding procedures, followed by recognizing appropriate linear linked
components. The algorithm is tuned to have either high precision or recall;
however, no single model has succeeded in achieving both simultaneously. Apart
from this, deep learning models have been used for contrail detection using
images of GOES-16 satellite[19] and images from Himawari-8 stationary
satellite[20]. However, apart from [21] that has used Hough Transforms to
segment AVHRR imagery, no work to our knowledge, has been done in over two
decades to implement neural network-based segmentation models for contrails
detection and Landsat-8 data.

This work explores the potential of ’deep convolutional nets’ to identify and
segment contrails in satellite imagery. For this purpose, we use popular
semantic segmentation models, such as UNet[22], PSP Net[23], DeepLab V3[24]
and DeepLab V3+[25] with different combinations of loss functions and encoders
to adjust it accordingly for capturing the intricacies of contrails and
achieve a more generalized result. We train these models on the labeled
Landsat-8 contrails dataset proposed by [2], which consists of several scenes
from the Landsat-8 satellite in which the contrails have been identified and
marked as bounding polygons. We hope our proposed methods can be used as a
benchmark for further work on this dataset and open the way for more
substantial research on this domain.

##  2 Data

The dataset[2] includes Landsat-8 scenes (primarily from 2018 and inside the
viewable extent of the GOES-16 satellite), which have been reviewed by human
labellers taught to identify and mark the bounding polygon of each contrail in
the scene. The dataset has 4289 scenes primarily from 2018 of the Landsat-8
satellite, out of which 47% of scenes have at least one contrail.

While prediction work has already been done on the GOES-16 and other NOAA
series satellite images, using images of a low-earth orbit satellite such as
Landsat-8 is beneficial for contrails as its images have a high spatial
resolution of 30m and 100m per pixel for cirrus and thermal infrared bands
respectively. This makes it easier to distinguish contrails from natural
cirrus clouds, which are made of ice crystals and look like long and thin
white lines. Each scene has a true-color RGB image, a false color image, and
labeled contrail points for this purpose.

We use the false colour images and the contrail labels by generating them as a
ground truth mask for each image. False colour images are generated by
extracting the brightness temperature difference or red channel between the 12
µm and 11 µm bands, the 1.37 µm cirrus cloud reflectance band, which is the
green channel and is omitted for nighttime images to avoid confusion and the
12 µm brightness temperature blue channel. The contrails in these images
appear as black linear clouds, making it easier for the models to
differentiate between cirrus clouds and contrails than RGB images for
segmentation.

##  3 Methodology

###  3.1 Dataset Preprocessing

The dataset is available in JSON string format stored in Google Cloud Storage.
There are 100 files, each with the Landsat-8 filename, polygon bounds of
contrails in the scene, and deidentified advected flight waypoints for each
labeled scene. We manually preprocessed the data by constructing the ground
truth mask from the polygon bounds using matplotlib collections[26]. We then
resized the images and the masks to 512x512 dimensions for training; anything
less than 512 led to a deterioration in results. We use only images with at
least one contrail for training due to class imbalance between contrails and
background and to help the model generalize better. The dataset has been
further divided into 80% for training (1737) and 20% for testing (434). We
opted to use the false color images as it was visually easier to identify the
contrail than RGB images. Experimentally as well, the models performed better
on false color images.

###  3.2 Models

We experimented with different types of state-of-the-art segmentation
networks, namely UNet[22], an encoder-decoder-based standard network, PSP
Net[23], a pyramid pooling module-based network, DeepLabV3[24] and
DeepLabV3+[25] which are a combination of both encoder-decoder and pyramid
pooling-based module. We used multiple backbones for each of these
architectures. We employed Resnet101[27],ResNeXt101-32x4d[28] and
Xception71[29] with ImageNet pretrained weights.We experimented with both
pretrained weights and random initialization and found that although results
were almost the same, using pretrained weights helped the models to converge
at least 30% faster. We chose these backbones for our segmentation models
as[30] showed them as most effective for semantic segmentation of fine lines.

Adam optimizer was used for all these instances with a learning rate of 0.0001
and sigmoid activation function. We tried various loss functions such as
Focal[31], Tversky[32], Focal-Tversky[33], Dice[34], and Jaccard loss[35].
Intersection over Union (IoU) was used as the primary evaluation metric. We
used different loss functions and noticed that an over-suppression of the
Focal-Tversky is observed when the class accuracy is high, as the model is
close to convergence. In contrast, for Dice loss, we noticed that it is very
unstable as the model goes closer to converges and hence doesn’t converge
well. To combat these shortcomings, we used a combination of Dice and Focal-
Tversky loss, as shown in equation. 1.

$\centering\begin{split}TotalLoss=&\sum_{c}\Bigg{(}\delta\left(1-\frac{\sum_{i=1}^{N}p_{ic}g_{ic}+\epsilon}{\sum_{i=1}^{N}p_{ic}+g_{ic}+\epsilon}\right)+\\\
&(1-\delta)\left(1-\frac{\sum_{i=1}^{N}p_{ic}g_{ic}+\epsilon}{\sum_{i=1}^{N}p_{ic}g_{ic}+\alpha\sum_{i=1}^{N}p_{i\bar{c}}g_{ic}+\beta\sum_{i=1}^{N}p_{ic}g_{i\bar{c}}+\epsilon}\right)^{1/\gamma}\Bigg{)}\end{split}\@add@centering$
(1)

N denotes the total number of pixels in an image, $g_{ic}$ and $p_{ic}$
represent the per pixel ground truth and predicted probability respectively
for contrail class c, similarly $g_{i\bar{c}}$ and $p_{i\bar{c}}$ represent
the non- contrail class $\bar{c}$. $\alpha$, $\beta$, and $\gamma$ are hyper-
parameters for Focal-Tversky loss that can be tuned. $\delta$ is a hyper-
parameter that decides the percentage of contribution of both Focal-Tversky
and Dice loss towards the final loss calculated. After careful
experimentation, we observed the best results with $\delta=0.5$.

Figure 1: Comparison of RGB, false colour, ground truth, and predicted mask
for test images

##  4 Results

Table 1 compares the train and test IoU of all the segmentation models-encoder
combinations. While there is about ±0.03 difference in the IoUs among all the
combinations, we notice that the combination of UNet[22] architecture with
Xception 71[29] backbone gives the best IoU of 0.4395. While the IoU was low,
the model produced satisfactory contrail masks. We attribute this to the fact
that the fundamental shape of the labels, i.e. thin and long hence IoU
calculation is affected drastically if the prediction is even a few pixels
off. We also tried simple data augmentation techniques, such as rotation and
flipping, which didn’t improve the IoU. Also, since we used false color images
for our purpose, using hue saturation augmentation caused deterioration in our
results.

We also noticed several mislabeled masks of contrails in the dataset. The
dataset has several noisy labels due to manual labelling, due to which it
suffers from large intra and inter-observer variability. Furthermore, numerous
contrail labels in the dataset are broader than the corresponding visible
contrail in the original image. This and a severe class imbalance in the
dataset were the two significant issues limiting the model’s ability. We have
used Dice and Focal-Tversky to create our loss function, as these are well
known for dealing with class imbalance issues. We chose to consider images
with at least one contrail for training. While these factors may affect the
IoU, our combinations still have performed convincingly on the test dataset,
evident in examples of the model’s predictions against the ground truth, as
provided in Figure 1

Table 1: IoU scores of segmentation models with different backbones |  | UNet | PSP Net | DeepLabV3 | DeepLabV3+
---|---|---|---|---|---
ResNet 101 | Training | 0.6479 | 0.5032 | 0.4714 | 0.7048
| Test | 0.3410 | 0.3788 | 0.4143 | 0.4015
ResNext 101-32x4d | Training | 0.5411 | 0.6500 | 0.7211 | 0.6657
| Test | 0.4224 | 0.4044 | 0.4339 | 0.4266
Xception 71 | Training | 0.6887 | 0.7272 | 0.7730 | 0.6290
| Test | 0.4395 | 0.4027 | 0.4246 | 0.4230

##  5 Conclusion

Contrails account for about 57% of the global warming caused by the aviation
industry[36]. Robust contrail detection and segmentation is the first step in
quantifying its lifetime impact and linking it to possible flights that may
have caused it, thus assisting in tracking and avoiding contrails.

To the best of our knowledge, we are the first to present a detailed work on
Landsat-8 data and evaluation on benchmarking of different state-of-the-art
models for semantic segmentation for contrail detection. Using false color
images of Landsat-8 imagery, we achieve 0.4395 testing IoU UNet architecture
with Xception 71 backbone. While the IoU is low, it fails to capture the
model’s true performance on the task. In our experiments, the model is more
often than not able to segment contrails proficiently.

While this work is a solid first step in utilizing deep learning methods for
contrail detection, there is much room for improvement. In future studies, we
would like to experiment with temperature-based colour preprocessing. To test
the model’s robustness, we want to calculate uncertainty estimates under
varying domains of satellite imagery and evaluate the model using a more
suitable metric instead of IoU. Contextual information can also be
incorporated based on publicly available datasets, such as overlapping
segmentation masks containing land cover, naturally occurring clouds, flight
paths etc. Another avenue for future experiments could be using self-
supervised and pseudo-labelling training techniques, attention-based models,
introduction of discriminator and vision transformers.

##  6 Acknowledgments

We would like to thank Mars Rover Manipal, an interdisciplinary student
project team of MAHE, for providing the necessary resources for our research.
We are grateful to our faculty advisor, Dr Ujjwal Verma, for providing the
necessary guidance.

## References

  * [1] Denis Avila, Lance Sherry, and Terry Thompson.  Reducing global warming by airline contrail avoidance: A case study of annual benefits for the contiguous united states.  Transportation Research Interdisciplinary Perspectives, 2:100033, 2019.
  * [2] Kevin McCloskey, Scott Geraedts, Christopher Van Arsdale, and Erica Brand.  A human-labeled landsat-8 contrails dataset.  In ICML 2021 Workshop on Tackling Climate Change with Machine Learning, 2021.
  * [3] Mannstein Vázquez-Navarro, Hermann Mannstein, and Stephan Kox.  Contrail life cycle and properties from 1 year of msg/seviri rapid-scan images.  Atmospheric Chemistry and Physics, 15(15):8739–8749, 2015.
  * [4] Lisa Bock and Ulrike Burkhardt.  Reassessing properties and radiative forcing of contrail cirrus using a climate model.  Journal of Geophysical Research: Atmospheres, 121(16):9717–9736, 2016.
  * [5] Ulrike Burkhardt and Bernd Kärcher.  Global radiative forcing from contrail cirrus.  Nature climate change, 1(1):54–58, 2011.
  * [6] Fred Pearce.  How airplane contrails are helping make the planet warmer.  YaleEnvironment360, 2019.
  * [7] Jarlath Molloy, Roger Teoh, Seán Harty, George Koudis, Ulrich Schumann, Ian Poll, and Marc EJ Stettler.  Design principles for a contrail-minimizing trial in the north atlantic.  Aerospace, 9(7):375, 2022.
  * [8] Hermann Mannstein, Peter Spichtinger, and Klaus Gierens.  A note on how to avoid contrail cirrus.  Transportation Research Part D: Transport and Environment, 10(5):421–426, 2005.
  * [9] Ulrich Schumann.  A contrail cirrus prediction model.  Geoscientific Model Development, 5(3):543–580, 2012.
  * [10] Thibaud M Fritz, Sebastian D Eastham, Raymond L Speth, and Steven RH Barrett.  The role of plume-scale processes in long-term impacts of aircraft emissions.  Atmospheric Chemistry and Physics, 20(9):5697–5727, 2020.
  * [11] Klaus Gierens, Sigrun Matthes, and Susanne Rohs.  How well can persistent contrails be predicted?  Aerospace, 7(12):169, 2020.
  * [12] Hermann Mannstein, Richard Meyer, and Peter Wendling.  Operational detection of contrails from noaa-avhrr-data.  International Journal of Remote Sensing, 20(8):1641–1660, 1999\.
  * [13] Richard Meyer, H Mannstein, R Meerkötter, U Schumann, and P Wendling.  Regional radiative forcing by line-shaped contrails derived from satellite data.  Journal of Geophysical Research: Atmospheres, 107(D10):ACL–17, 2002\.
  * [14] Rabindra Palikonda, Patrick Minnis, David P Duda, and Hermann Mannstein.  Contrail coverage derived from 2001 avhrr data over the continental united states of america and surrounding areas.  Meteorologische Zeitschrift, 14(4):525–536, 2005.
  * [15] Patrick Minnis, Rabindra Palikonda, Bryan J Walter, J Kirk Ayers, and Hermann Mannstein.  Contrail coverage over the north pacific from avhrr data.  Meteor. Z, 2005.
  * [16] Richard Meyer, Rüdiger Buell, Christian Leiter, Hermann Mannstein, Susanna Pechtl, Taikan Oki, and Peter Wendling.  Contrail observations over southern and eastern asia in noaa/avhrr data and comparisons to contrail simulations in a gcm.  International Journal of Remote Sensing, 28(9):2049–2069, 2007\.
  * [17] Mannstein Vazquez-Navarro, H Mannstein, and B Mayer.  An automatic contrail tracking algorithm.  Atmospheric Measurement Techniques, 3(4):1089–1101, 2010.
  * [18] David P Duda, Patrick Minnis, Konstantin Khlopenkov, Thad L Chee, and Robyn Boeke.  Estimation of 2006 northern hemisphere contrail coverage using modis data.  Geophysical Research Letters, 40(3):612–617, 2013.
  * [19] Luke Kulik.  Satellite-based detection of contrails using deep learning.  PhD thesis, Massachusetts Institute of Technology, 2019.
  * [20] Guoyu Zhang, Jinglin Zhang, and Jian Shang.  Contrail recognition with convolutional neural network and contrail parameterizations evaluation.  SOLA, 14:132–137, 2018.
  * [21] John M Weiss, Sundar A Christopher, and Ronald M Welch.  Automatic contrail detection and segmentation.  IEEE transactions on geoscience and remote sensing, 36(5):1609–1619, 1998.
  * [22] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [23] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  * [24] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [25] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [26] J. D. Hunter.  Matplotlib: A 2d graphics environment.  Computing in Science & Engineering, 9(3):90–95, 2007.
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [28] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  arXiv preprint arXiv:1611.05431, 2016.
  * [29] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
  * [30] Rongyu Zhang, Lixuan Du, Qi Xiao, and Jiaming Liu.  Comparison of backbones for semantic segmentation network.  In Journal of Physics: Conference Series, volume 1544, page 012196\. IOP Publishing, 2020.
  * [31] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection, 2017.
  * [32] Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour.  Tversky loss function for image segmentation using 3d fully convolutional deep networks, 2017.
  * [33] Nabila Abraham and Naimul Mefraz Khan.  A novel focal tversky loss function with improved attention u-net for lesion segmentation, 2018.
  * [34] Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso.  Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations.  In Deep learning in medical image analysis and multimodal learning for clinical decision support, pages 240–248. Springer, 2017.
  * [35] David Duque-Arias, Santiago Velasco-Forero, Jean-Emmanuel Deschaud, Francois Goulette, Andrés Serna, Etienne Decencière, and Beatriz Marcotegui.  On power jaccard losses for semantic segmentation.  In VISAPP 2021: 16th International Conference on Computer Vision Theory and Applications, 2021.
  * [36] D.-S. Lee, David W. Fahey, Agnieszka Skowron, Myles R. Allen, Ulrike Burkhardt, Qi Chen, Sarah J. Doherty, Sarah Freeman, Piers M. Forster, Jan S. Fuglestvedt, Andrew Gettelman, Rubén Rodríguez De León, Ling L. Lim, Marianne Tronstad Lund, Richard J. Millar, Bethan Owen, J. E. Penner, Giovanni Mario Pitari, Michael J. Prather, Robert Sausen, and Laura J. Wilcox.  The contribution of global aviation to anthropogenic climate forcing for 2000 to 2018.  Atmospheric Environment (Oxford, England : 1994), 244:117834 – 117834, 2020.