# PointFlow: Flowing Semantics Through Points for Aerial Image Segmentation

Xiangtai Li1, Hao He2,3∗, Xia Li4, Duo Li5, Guangliang Cheng6,
Jianping Shi6, Lubin Weng2, Yunhai Tong1, Zhouchen Lin1
1 Key Laboratory of Machine Perception (MOE), Peking University
2 NLPR, Institute of Automation, Chinese Academy of Sciences
3 School of Artificial Intelligence, University of Chinese Academy of Sciences
4 ETH Zurich 5 HUKST 6 SenseTime Research The first two authors contribute
equally. Email<EMAIL_ADDRESS>
###### Abstract

Aerial Image Segmentation is a particular semantic segmentation problem and
has several challenging characteristics that general semantic segmentation
does not have. There are two critical issues: The one is an extremely
foreground-background imbalanced distribution, and the other is multiple small
objects along with the complex background. Such problems make the recent dense
affinity context modeling perform poorly even compared with baselines due to
over-introduced background context. To handle these problems, we propose a
point-wise affinity propagation module based on the Feature Pyramid Network
(FPN) framework, named PointFlow. Rather than dense affinity learning, a
sparse affinity map is generated upon selected points between the adjacent
features, which reduces the noise introduced by the background while keeping
efficiency. In particular, we design a dual point matcher to select points
from the salient area and object boundaries, respectively. Experimental
results on three different aerial segmentation datasets suggest that the
proposed method is more effective and efficient than state-of-the-art general
semantic segmentation methods. Especially, our methods achieve the best speed
and accuracy trade-off on three aerial benchmarks. Further experiments on
three general semantic segmentation datasets prove the generality of our
method. Code will be provided in (https://github.com/lxtGH/PFSegNets).

##  1 Introduction

High spatial resolution (HSR) remote sensing images contain various geospatial
objects, including airplanes, ships, vehicles, buildings, etc. Understanding
these objects from HSR remote sensing imagery has great practical value for
urban monitoring and management. Aerial Image segmentation is an important
task in remote sensing understanding that can provide semantic and
localization information cues for interest targets. It is a specific semantic
segmentation task that aims to assign a semantic category to each image pixel.

Figure 1:  Illustration of an aerial image segmentation example and our
proposed module. The first row presents the input image and ground truth with
complex backgrounds and small objects. The second row indicates the schematic
diagram on dense affinity propagation and our proposed point-based propagation
module.

However, besides the large scale variation problems in most semantic
segmentation datasets [12, 70, 40, 6], aerial images have their own
challenging problems including high background complexity [69], background and
foreground imbalance [50], tiny foreground objects in high resolution images.
As shown in the first row of Fig.1, the red boxes show the tiny objects in the
scene while the yellow box and blue box show the complex background context,
including houses and trees receptively. Current general semantic segmentation
methods mainly focus on scale variation in the natural scene by building
multi-scale feature representation [68, 8] or enhancing the object boundaries
with specific design module [23, 45]. They fail to work well due to the lack
of explicit modeling for the foreground objects. For example, several dense
affinity-based methods [16, 59] also obtain inferior results mainly because
the imbalanced and complex background will fool the affinity learning on small
objects. For example, both yellow boxes and blue boxes have the same semantic
meaning of background but with a huge appearance change. Dense affinity
learning forces pixels on small objects to absorb such noisy context which
leads to inferior segmentation results. FarSeg [69] adopts FPN-like [35]
design and solve the background and foreground imbalance problems by
introducing a foreground-aware relation module. However, for small objects,
there still exist some semantic gaps in different features in FPN. Namely, the
gap is between the high-resolution features with low semantic information and
low-resolution features with high semantic information. As shown in Fig.1,
tiny objects like cars need more semantic information in lower layers with
high resolution.

In this paper, we propose a point-based information propagation module to
handle the previous problems stated above. We propose PointFlow Module (PFM),
a novel and efficient module for specific semantic points propagation between
adjacent features. Our module is based on the FPN framework [35, 22] to bridge
the semantic gap. As shown in the last row of Fig.1, rather than simple fusion
or dense affinity propagation on each point as the previous work non-local
module [48], PointFlow selects the several representative points between any
adjacent feature pyramid levels. In particular, we design Dual Point Matcher
by selecting matched point features from the salient area and object
boundaries receptively. The former is obtained from explicit max pooling
operation on the learned salient map. The latter is conditioned on the
predicted object boundaries where we adopt a subtraction-based prediction.
Then the point-wise affinity is estimated according to the point features that
are sampled from both adjacent features. Finally, the higher layer points are
fused into lower layers according to the affinity map. Our PFMs select and
propagate points on foreground objects and sampled background areas to
simultaneously handle both the semantic gap and foreground-background
imbalance problem.

Then we carry out detailed studies and analysis on PFM in the experiment part,
where it improves the various methods by a large margin with negligible GFlops
increase. Based on the FPN framework, by inserting PFMs between feature
pyramids, we propose the PFNet. In particular, PFNet surpasses the previous
method FarSeg [69] by 3.2% point on iSAID [50]. Moreover, we also benchmark
the recent state-of-the-art general semantic segmentation methods [47, 59, 31]
on three aerial segmentation datasets including iSAID, Vaihingen and Postdam
for the community. Benefited from efficient FPN design [22], our PFNet also
achieves the best speed and accuracy on three benchmarks. Finally, we further
verify the effectiveness of PFM on general semantic segmentation benchmarks,
including Cityscapes [12], ADE-20k [70], and BDD [56] and it achieves
considerable results with previous work [8, 59] with fewer GFLOPs. Our main
contributions are three-fold:

1) We propose PointFlow Module (PFM), a novel and efficient module for poise-
wised affinity learning, and we design a Dual Point Matcher to select the
matched sparse points from salient areas and boundaries in a complementary
manner.

2) We append PFM into FPN architecture and build a pyramid propagation network
called PFNet.

3) Extensive experiments and analysis indicate the efficacy of PFM. We
benchmark 15 state-of-the-art general segmentation methods on three aerial
benchmarks. Our PFNet achieves state-of-the-art results on those benchmarks
also with the best speed and accuracy trade-off. We further prove the
generality of our method on three general semantic segmentation datasets.

##  2 Related Work

General Semantic Segmentation The general semantic segmentation has been
eminently motivated by the fully-convolutional networks (FCNs) [37]. The
following works [68, 7, 8, 9, 53, 47, 30] mainly exploit the spatial context
to overcome the limited receptive field of convolution layer which leads to
the multi-scale feature representation. For example, ASPP [8] utilizes atrous
convolutions [57] with different atrous rate to extract features with the
different receptive field, while PPM [68] generates pyramidal feature maps via
pyramid pooling. Several work [43, 2, 51, 67, 4, 22, 25] use the encoder-
decoder architecture to refine the output details. Recent works [31, 59, 20,
62, 64, 72, 58, 10, 32, 65, 66, 26, 54, 28] propose to use non-local-like
operators or losses [46, 49, 24, 55] to harvest the global context of input
images. Meanwhile, several works [23, 45, 60, 27] propose to refine the object
boundaries via specific designed processing. These general semantic
segmentation methods ignore the special issues including imbalanced
foreground-background pixels for modeling the context and increased small
foreground objects in the Aerial Imagery. Thus these methods get inferior
results which will be shown in the next section.

Semantic Segmentation of Aerial Imagery Several earlier works [21, 39, 38]
focus on using multi-level semantic features on local patterns of images using
deep CNN. Also, there exist a lot of applications, such as land use [19],
building or road extraction [14, 33, 52, 3], agriculture vision [11]. They
design specific methods based on existing semantic segmentation methods for
special application scenario. In particular, relation net [41] captures long-
range spatial relationships between entities by proposing spatial and channel
relation modules. Recently, FarSeg [69] propose relation-based and
optimization-based foreground modeling to handle the foreground-background
imbalance problems in remote sensing imagery. However, the missing explicit
exploration of semantics propagation between adjacent features limits the
performance on the segmentation of small objects.

Multi Scale Feature Fusion Based on the FPN framework [35], rather than simple
top-down additional fusion, several works propose to fuse feature through
gates [15, 25], neural architecture search [17], pixel-level alignment [29] or
adding bottom up path [36], dense affinity learning propagation[61]. Such full
fusion methods may emphasize background objects like roads where the imbalance
problem exists widely in aerial images. Our proposed PFM follows the design of
FPN by propagating the semantics from the top to bottom. In contrast, rather
than full fusion like previous works, our methods are based on point-level
which select the several representative points to overcome the pixel imbalance
problems in aerial imagery and leads to better results.

##  3 Method

In this section, we will first introduce some potential issues on dense point
affinity learning for aerial segmentation task. Then we will provide detailed
descriptions of our PointFlow module(PFM) to resolve the issues by selecting
key semantic points for propagation efficiently. Finally, we will present our
PFNet for aerial imagery segmentation.

Method | OS | mIoU | $\Delta$
---|---|---|---
dialated FCN[37, 57](baseline) | 8 | 59.0 | -
DAnet[68] | 8 | 30.3 | 28.7$\downarrow$
OCnet(ASP-OC) [59] | 8 | 40.2 | 18.8$\downarrow$
DAnet+FPN [35] | 8 | 59.3 | 0.3$\uparrow$
DAnet+our PFNet decoder | 8 | 65.6 | 6.6$\uparrow$
SemanticFPN [22](baseline) | 32 | 61.3 | -
+dense affinity [61] | 32 | 58.9 | 2.4$\downarrow$
+our PFM | 32 | 65.0 | 3.7$\uparrow$

Table 1:  Simple experiment results on iSAID validation dataset. The dense
affinity results in inferior results over various baselines. Appending our
proposed PointFlow module results in a significant gain. OS: Output Stride in
backbone.

###  3.1 Preliminary

Recent dense affinity based methods [48, 59, 16, 46, 61] have shown
progressive results for semantic segmentation. The core idea of these methods
is to model the pixel-wised relationship to harvesting the global context. As
shown in Equ. 1, in the view of self-attention [46], each pixel $p$ in 2-D
input feature $F$ $\in\mathbb{R}^{C\times H\times W}$ is connected to all the
other pixels to calculate the pixel-wised affinity where $A$ is the affinity
function and it outputs affinity matrix $\in\mathbb{R}^{HW\times HW}$. $C$,
$H$, and $W$ denote the channel dimension, height, and width, respectively.
Note that definitions of $A$ can be different; we use the same label for
simplicity.

$F^{r}(p)=A(F(p),F(p)){F(p)}$ (1)

However, applying these methods directly on the iSAID dataset leads to
inferior results even compared with various baseline methods, as shown in Tab.
1 whether such module is appended after FCN backbone or is inserted into
feature pyramids. The reason has two folds: (1) There exist extremely
imbalanced foreground-background objects in the iSAID dataset. Explicit
affinity modeling on complex background brings noise for outputs. (2) Too many
small objects exist on the iSAID dataset, which requires high resolution and
high semantic representation.

To solve the first problem, rather than dense affinity modeling, we can use a
point sampler $\beta$ to select matched representative points $\hat{p}$ to
balance the background context ratio while keeping efficiency. For the second
problem, to fill the semantic gap on small objects, we adopt the FPN framework
and change the inputs of $A$ by using adjacent features in a top-down manner
shown in Equ. 2:

$F^{r}(\hat{p})=A(\beta(F_{l}(\hat{p})),\beta(F_{l-1}(\hat{p})))\beta(F_{l}(\hat{p})))$
(2)

where $F_{l}$ and $F_{l-1}$ are adjacent features in the FPN framework and
$\hat{p}$ is sampled pixels for affinity modeling. We will detail the $\beta$
in the following part. As shown in Tab.1, our method improves the baselines by
a significant margin.

###  3.2 PointFlow Module

Motivation and Overview As the previous section shows the limitation of dense
affinity on aerial image segmentation, we argue that unnecessary background
pixels context may bring noises for foreground objects. Considering this, we
propose to propagate context information through selective points, which can
keep the efficiency in both speed and memory. Meanwhile, the semantic gap
problems can also be fixed after propagation leading to high-resolution
feature representation with high semantics, which is why we adopt FPN-
framework design [35] in a top-down manner. Since our framework works in a
top-down manner, and the semantics flow into low-level features through
points, we name our module PointFlow. Our PointFlow is built on the FPN
framework [35], where the feature map of each level is compressed into the
same channel depth through two 1$\times$1 convolution layers before entering
the next level. Our module takes two adjacent feature maps as inputs
$F_{l-1}\in\mathbb{R}^{C\times H\times W}$ and $F_{l}\in\mathbb{R}^{C\times
H/2\times W/2}$ as the inputs where $l$ means the index of feature pyramid and
output refined $F_{l-1}^{r}\in\mathbb{R}^{C\times H\times W}$. For modeling
$\beta$, we propose the Dual Point Matcher to select the points, and then the
point-wise affinity can be calculated between adjacent points. Finally, the
points with high-resolution and low semantics can be enhanced by the points
with low-resolution high semantics according to the estimated affinity map.
The process is shown in Fig. 2(a).

Figure 2:  (a), The overal pipeline of our proposed PointFlow Module. Left:
Two adjacent features with one salient map are sent to the Dual Index
Generator to obtain the sampled indexes. Right: The sampled point features are
propagated from top to the bottom and finally scattered into the low level
features point-wisely. (b), The detailed operation on proposed Dual Index
Generator. (c), We design the PF Network Architecture by inserting PF modules
into FPN-like framework.

Dual Point Matcher The critical issue is how to find the corresponding points
between two adjacent maps. We argue that most salient areas can be represented
as key points for balanced pixel-level propagation due to the unbalanced
pixels between the foreground and background. Meanwhile, since there are many
small objects in aerial scenes that need more fine-grained location cues, the
boundary areas can also be considered the key points. Thus we design a novel
Dual Point Matcher to consider the most salient part of inputs and object
boundaries at the same time. The Dual Point Matcher has two steps: (1)
Generate the salient map. (2) Generate sampled indexes from Dual Index
Generator.

For the first step, we combine the input feature maps where the high-
resolution part $F_{l-1}$ is downsampled into the same low resolution through
bilinear interpolation. The resized feature is denoted as
$\widetilde{F}_{l-1}$. Then we perform one $3\times 3$ convolution following
with sigmoid function to generate the saliency map $M_{l}$. The process is
shown as follows:

$M_{l}=Sigmoid(\text{conv}_{l}(\text{Concat}(F_{l},\widetilde{F}_{l-1}))),$
(3)

For the second step, we take $F_{l}$ and $M_{l}$ as the inputs of the Dual
Index Generator. We perform the adaptive max pooling on such map to obtain the
most salient points. To highlight the salient part of foreground objects, we
multiply such map on $F_{l}$ with residual design as attention map shown in
Equ. 4:

$F_{l}^{s}=MaxPool(M_{l})\times F_{l}+F_{l},$ (4)

We simply choose the salient indexes from $MaxPool(M_{l})$. $K$ is the number
of pooled points, and it equals to the product of adaptive pooling kernels. We
denote the salient indexes as $I(s)$ for short.

For boundary point selection, rather than simply using the binary supervision
on the input feature $F_{l}$ or $F_{l-1}$ for boundary prediction, we propose
to adopt residual prediction on the $F_{l}$. Our method is motivated by
Laplacian pyramids in image processing [1, 5]. In Laplacian pyramids, the edge
part of original images can be obtained by subtracting the smoothed upsampled
images. Motivated by that, we use the average pooling on saliency map $M_{l}$
and multiply the pooled map on $F_{l}$ for smoothing inner content, then we
subtract the such smoothed part from $F_{l}$ to generate the sharpened feature
$\widetilde{F}_{l}^{b}$ for boundary prediction. The process is shown in Equ.
5:

$\widetilde{F}_{l}^{b}=F_{l}-AvgPool(M_{l})\times F_{l},$ (5)

After the boundary prediction using $\widetilde{F}_{l}^{b}$, we obtain the
boundary map $B_{l}$. Following the previous step, we simply sample Top-K
points from the edge maps(K=128 by experiment) according to their confidence
scores. We denote the boundary indices $I(b)$ for short. In total, the Dual
Index Generator samples the key points in an orthogonal way by selecting
points from specific regions according to the salient map $M_{l}$. The total
process of Dual Index Generator is shown in Fig. 2(b).

Dual Region Propagation After the point matcher, we obtain the indexes $I(s)$
and $I(b)$, respectively. Then we sample the points from map from salient
feature $F_{l}^{s}$ and original input feature $F_{l-1}$. For each selected
point, a point-wise feature representation is extracted on both adjacent input
features. Note that features $f$ for a real-value point are computed by
bilinear interpolation of 4 nearest neighbors that are on the regular grid. We
use normalized girds during the implementation. We denote $f_{l}^{s}$ and
$f_{l}^{b}$ as sampled feature point at stage $l$ for salient part and
boundary part. We propagate those sampled points independently. For each
sampled point $\hat{p}$, the top-down propagation process is shown in Equ 6.

$f_{l-1}(\hat{p})^{r}=\sum_{i\in{\\{I(b),I(s)\\}}}A(f_{l-1}^{i}(\hat{p}),f_{l}^{i}(\hat{p}))f_{l}^{i}(\hat{p})+f_{l-1}^{i}(\hat{p}),$
(6)

where $A$ is affinity function, $i$ means the indexes whether from $I(s)$ or
$I(b)$. For $A$, we use the point-wise matrix multiplication along with
softmax function for normalization. Following the previous work [18], we adopt
the residual design for easier training. We calculate the sampled high
semantic points through point-wise affinity according to the semantic
similarity on sampled points with low semantics, which avoids the redundant
background information in the aerial scene. Since we propagate semantics two
times independently, we term two flows as salient point flow and boundary
point flow, respectively. Finally, the refined feature $F_{l-1}^{r}$ is
obtained by scattering the $f_{l-1}^{r}$ into $F_{l-1}$ according to the
indices $I(s)$ and $I(b)$.

###  3.3 Network Architecture

Overview Fig. 2 illustrates the our network architecture, which contains a
bottom-up pathway as the encoder and a top-down pathway as the decoder. The
encoder is backbone network with multiple feature pyramid outputs while the
decoder is lightweight FPN equipped with our PFMs.

Network Architecture The encoder uses the ImageNet pretrained backbone with OS
32 rather dilation stragety with OS 8 for efficient inference. We additionally
adopt the Pyramid Pooling Module (PPM) [68] for its superior efficiency and
effectiveness to capture contextual information. In our setting, the output of
PPM has the same resolution as that of the last stage. PFNet decoder takes
feature maps from the encoder and uses the refined feature pyramid for final
aerial segmentation according to previous work design [69, 22]. By simply
replacing normal bilinear up-sampling with our PF module in top-down pathway
of FPN, the PFNet decoder finally concatenates all the refined
$F_{l}^{r}$(where $l$ ranges from 2 to 5) by upsampling the inputs to the same
resolution(1/4 resolution of input) and perform prediction. Note that our
module can also be integrated into other architectures including Deeplabv3 [8]
with a slight modification by appending such decoder after its head. More
details can be found in the experiment part.

Loss Function For edge prediction in each PFM, we adopt binary BCE loss
$L_{bce}$. For final segmentation prediction, we adopt the cross-entropy loss.
The two losses are weighted to 1 by default.

##  4 Experiments

Overview: We will firstly perform ablation studies on iSAID dataset and give
detailed analysis and comparison on PFM. Then we benchmark several recent
works on Vaihingen and Potsdam datasets. Finally, we prove the
generalizability of our module on general segmentation datasets.

###  4.1 Aerial Image Segmentation

DataSets: We use iSAID [50] dataset for ablation studies and report results on
remaining datasets. iSAID [50] consists of 2,806 HSR images. The iSAID dataset
provides 655,451 instances annotations over 15 categories of the object and it
is the largest dataset for instance segmentation in the HSR remote sensing
imagery. We also use Vaihingen and Postdam
datasets111https://www2.isprs.org/commissions/comm2/wg4/benchmark/ for
benchmarking.

Implementation detail and Metrics: We adopt ResNet-50 [18] by default.
Following the same setting [69], for all the experiments, these models are
trained with 16 epoch on cropped images. For data augmentation, horizontal and
vertical flip, rotation of 90 · k (k = 1, 2, 3) degree were adopted during
training. For data preprocessing, we crop the image into a fixed size of (896,
896) using a sliding window striding 512 pixels. We use the mean intersection
over union (mIoU) as the main metric for object segmentation to evaluate the
proposed method if not specified. The baseline for ablation studies is
Semantic-FPN [22] with OS 32.

+PPM | +salient point flow | \+ boundary point flow | mIoU(%)
---|---|---|---
- | - | - | 61.3
✓ | - | - | 63.8
| ✓ | ✓ | 65.0
✓ | ✓ | - | 64.8
✓ | - | ✓ | 66.2
✓ | ✓ | ✓ | 66.9

(a) Effect of dual flow propagation on baseline.

Method | $\hat{\mathbf{F}_{3}}$ | $\hat{\mathbf{F}_{4}}$ | $\hat{\mathbf{F}_{5}}$ | mIoU(%)
---|---|---|---|---
Baseline+PPM |  |  |  | 63.8
| ✓ |  |  | 65.8
|  | ✓ |  | 65.6
|  |  | ✓ | 65.5
|  | ✓ | ✓ | 66.5
| ✓ | ✓ | ✓ | 66.9

(b) Effect of Insertion Position. $\hat{\mathbf{F}_{l}}$ means the position
between $\mathbf{F}_{l}$ and $\mathbf{F}_{l}-1$.

Settings | mIoU(%)
---|---
baseline+PPM | 63.8
+DCNv1 [13] | 65.2
+DCNv2 [71] | 65.6
+desne affinity flow [49] | 62.0
+FAM [29] | 65.7
\+ Ours | 66.9

(c) Comparison with Other Propagation Methods.

Sampling Method | mIoU(%)
---|---
baseline+PPM | 63.8
uniform random | 64.0
attention based | 64.2
Our max pooling | 64.8

(d) Effect of salient point sampling in Dual Index Generator.

Settings | mIoU(%)
---|---
baseline + PPM | 63.8
top-down(td) | 66.9
bottom-up(bu) | 47.3
td then bu | 54.5

(e) Effect of propagation direction.

Settings | mIoU(%)
---|---
baseline+PPM | 63.8
direct prediction | 65.7
addition prediction | 65.5
Our subtraction based | 66.2

(f) Effect of edge generation module in Dual Index Generator.

Network | Backbone | mIoU(%) | GFlops
---|---|---|---
Deeplabv3 [8] | ResNet50 | 60.4 | 168.4
Deeplabv3 [8] | ResNet101 | 61.5 | 264.1
+FPN | ResNet50 | 62.3 | 183.4
+PF decoder | ResNet50 | 65.6 | 185.2
CCNet [20] | ResNet50 | 58.3 | 206.5
+FPN | ResNet50 | 60.2 | 220.8
+PF decoder | ResNet50 | 65.3 | 223.2

(g) Application on Other Architectures.

Table 2: Ablation studies. We first verify the effect of each module and
comparison results in the first row. Then we verify several design choices and
generality of our module in the second row.

Effectiveness on baseline models: In Tab. 2(a), adopting our PFMs leads to
better results than appending PPM [68] shown in both 2nd and 3rd rows with
about 1.2 % gap. After applying both PPM and PFM, there is a significant gain
over the baseline models shown in the last row. Only applying boundary flow is
slightly better than applying salient point flow which indicates the small
object problems are more severe than foreground-background imbalance problems
in this dataset. In Tab. 2 (b), we explore the effect on insertion position
with our PFMs. From the first three rows, PF improves all stages and gets the
greatest improvement at the first stage, which shows that the semantic gap is
more severe for small objects in lower layers. After appending all FPMs, we
achieve the best result shown in the last row.

Comparison with feature fusion methods: Tab. 2(c) give several feature fusion
methods [13, 29, 61] used on scene understanding tasks. For all the methods,
we replace these modules into the same position on PFnet decoder as in Fig
2(c) for fair comparison. Compared with DCN-like methods [13, 71, 29], our
method leads to significant gain over them since our method can better handle
the foreground semantics propagation.

Ablation on design choices: We give more detailed design studies in the second
row of Tab. 2. Tab. 2(d) explores several sampling methods for salient points
sampling. Attention based method is directly selecting top-K(K=128) points
from $M_{l}$ while uniform random sample is done by randomly selecting one
pixels from $7\times 7$ neighbor region of $M_{l}$(We report average result of
10 times experiments). Our max pooling based methods work the best among them.
Tab. 2(e) shows the propagation direction of PFM. Adding bottom up fusing
leads to bad results mainly because more background context is introduced into
the head which verifies our motivation of flowing semantics into the bottom.
Tab. 2 (f) shows the effect results on edge prediction. Our subtraction based
prediction has better results mainly due to better boundary prediction. This
is also verified in Tab. 3.

Figure 3:  Ablation studies on the number of sampled point for both point
flows. Best view it in color and zoom in.

Ablation on Number of Sampled Points: We first verify the best number of
sampled points on boundary point flow in Fig. 3(a) by increasing the number of
sampled pixels where we find the best number is 128. Sampling more points
leads to inferior results which indicates missing background context is also
important. Appending the boundary flow as the strong baseline, we explore the
kernel size of salient point flow where we find the best kernel size 14
$\times$ 14 (256 points in total) in Fig. 3(b). After selecting more points
(24$\times$24, 576 points in total), the performance drops a lot since the
imbalance problems exist. This verifies the same conclusion that the dense
affinity leads to bad results.

Application on Various Methods: Our PFM can be easily adopted into several
existing networks by extending PFNet decoder (shown in Fig. 2(c) yellow box)
after their heads. More details can be referred to supplementary. In Tab.
2(g), we verify two work including Deeplabv3[8] and CCNet [20] where we obtain
significant gains over these baselines. This proves the generalization of our
methods. Our method outperforms ResNet101-based models which indicates the
improvement is not obtained by extra parameters introduced by PFM.

Method | mIoU | F1(12px) | F(9px) | F1(5px) | F1(3px)
---|---|---|---|---|---
baseline+PPM | 63.8 | 88.2 | 86.2 | 85.6 | 84.3
+salient point flow: | 64.8 | 88.9 | 88.1 | 87.0 | 85.4
+boundary point flow | 66.2 | 93.2 | 91.2 | 89.0 | 88.4
+both | 66.9 | 94.2 | 93.2 | 90.2 | 89.0
direct prediction | 65.7 | 89.6 | 87.5 | 86.4 | 85.8
subtraction prediction | 66.2 | 93.2 | 91.2 | 89.0 | 88.4

Table 3: Ablation study on semantic boundaries where we adopt 4 different
thresholds for evaluation.

Effectiveness on Segmentation Boundaries: We further verify the boundary
improvements using F1-score metric [42] with different pixel thresholds in
Tab. 3. Appending boundary point flow leads to more significant improvements
than salient point flow due to the explicit supervision and propagation on
boundary pixels. Adopting both flows leads to the best results and it
indicates the complemented property of our approach. Moreover, as shown in the
last row of Tab. 3, our subtraction based edge prediction results better than
direct prediction where it has better mask boundary. We include boundary
prediction results in supplementary.

Figure 4:  Visualization of sampled points for both point flows. Top: Salient
Flow points. Bottom: Boundary Flow points. Best view it on screen.

Balanced foreground-background Points: We analyze the ratio of sampled points
on fore-ground parts over total sampled points by adding all three PFMs using
validation images. Compared with baseline unbalanced points with 2.89%
computed by ground truth mask, our method improves the ratio on foreground to
7.83% during the inference which resolves the problems of imbalanced points
aggregation. We use the best model found in ablation part.

Visualization of Sampled Points: In Fig. 4, we show several visual examples on
sampled points on the original images. The first row gives the salient point
results while the second row shows the boundary point results. We visualize
the points from the PFM in the last stage. As shown in Fig. 4, the salient
points uniformly locate around the foreground objects and several of them are
on the background sparsely. The boundary points are mainly on the boundary of
large foreground objects and the inner regions on small objects because the
downsampled feature representation makes it hard to predict small object
boundaries. More visual examples can be found in supplementary.

Benchmarking recent works on aerial images datasets: Recent work FarSeg [69]
reports results of several segmentation methods [68, 22, 8] on iSAID datasets.
We extend more representative work [20, 16, 31, 23] on iSAID, Vaihingen and
Postdam datasets under the same experiment setting. Note that, for all
methods, we use ResNet50 as backbone for fair comparison except for HRNet
[47]. The work [41] also reports results on Vaihingen and Postdam using weak
VGG-backbone [44]. Due to the lack of comparison with recent work, we re-
implement this method using ResNet50 backbone and trained on larger cropped
images and report mIoU as metric.222We will opensource all the models and code
for further research. All the methods use the single scale inference on
cropped images for testing.

Method | Backbone | mIoU | OS
---|---|---|---
DenseASPP [53] | RenNet50 | 57.3 | 8
Deeplabv3 [8] | ResNet50 | 60.4 | 8
Deeplabv3+ [9] | ResNet50 | 61.2 | 8
RefineNet [34] | ResNet50 | 60.2 | 32
PSPNet [68] | ResNet50 | 60.3 | 8
OCNet-(ASP-OC) [59] | ResNet50 | 40.2 | 8
EMANet [31] | ResNet50 | 55.4 | 8
CCNet [20] | ResNet50 | 58.3 | 8
EncodingNet [63] | ResNet50 | 58.9 | 8
SemanticFPN [22] | ResNet50 | 62.1 | 32
UPerNet [22] | ResNet50 | 63.8 | 32
HRNet[51] | HRNetW18 | 61.5 | 4
SFNet[29] | ResNet50 | 64.3 | 32
GSCNN[45] | ResNe50 | 63.4 | 8
RANet[41] | ResNet50 | 62.1 | 8
FarSeg [69] | ResNet50 | 63.7 | 32
PFNet | ResNet50 | 66.9 | 32

Table 4: Comparison with the state-of-the-art results on iSAID dataset.

Comparison with the state-of-the-arts on iSAID: We first benchmark more
results on iSAID dataset in Tab. 4 and then compare our PFNet with previous
work. Our PFNet achieves the state-of-the-art result among all previous work
by a large margin. Our method outperforms previous state-of-the-art FarSeg
[69] by 3.2%.

Experiments on Vaihingen and Potsdam: Rather than the previous work [41]
cropping the images into small patches, we adopt large patches as the iSAID
dataset and use more validation images for testing. That makes the
segmentation more challenging. The details of train and validation splitting
can be found in the supplementary. For the Vaihingen dataset, we preprocess
the images by cropping into 768$\times$ 768 patches. We adopt the same
training setting with iSAID dataset except for 200 epochs and larger learning
rate with 0.01. For the experiments on the Potsdam dataset, the images are
cropped into 896$\times$896 patches. The total training epoch is set to 80
with the initial learning rate of 0.01. As shown in Tab. 5, we benchmark
recent segmentation methods with two metrics including mIoU and mean-$F_{1}$.
Our PFNet achieves state-of-the-art results on two benchmarks.

Figure 5:  Speed (Inference Time) versus Accuracy (mIoU) on three aerial
segmentation datasets. The radius of circles represents the number of
parameters. All the methods are tested with one V-100 GPU card for fair
comparison. Our PFNet achieves the best speed and accuracy trade-off on three
benchmark. Real time is within 50ms. Best view it on screen and Zoom in.
Figure 6: Visual results on iSAID validation set. Compared with previous
works, our method obtains better segmentation results. Best view on the
screen. More visual results can be found in supplementary.

Efficiency Comparison: In Fig. 5, we further benchmark the speed and
parameters of our methods on above datasets. Compared with previous work,
PFNet achieves the best speed and accuracy trade-off on those three benchmarks
with fewer parameters without bells and whistles. Note that PFNet can also run
in real-time setting and also achieves a significant margin compared with
previous real-time methods[22, 29, 69].

Visual Results Comparison: In Fig. 6, we compare our method results with
several state-of-the-art methods [9, 23, 22] on the iSAID validation set. Our
PFNet has better segmentation results on handling false positives of small
objects and has fine-grain object mask boundaries.

Method | mIoU | mean-$F_{1}$ | mIoU | mean-$F_{1}$
---|---|---|---|---
PSPNet [68] | 65.1 | 76.8 | 73.9 | 83.9
FCN [37] | 64.2 | 75.9 | 73.1 | 83.1
OCnet(ASP-OC) [59] | 65.7 | 77.4 | 74.2 | 84.1
Deeplabv3+ [9] | 64.3 | 76.0 | 74.1 | 83.9
DAnet [16] | 65.3 | 77.1 | 74.0 | 83.9
CCnet[20] | 64.3 | 75.9 | 73.8 | 83.8
SemanticFPN [22] | 66.3 | 77.6 | 74.3 | 84.0
UPerNet [51] | 66.9 | 78.7 | 74.3 | 84.0
PointRend [23] | 65.9 | 78.1 | 72.0 | 82.7
HRNet-W18 [47] | 66.9 | 78.2 | 73.4 | 83.4
GSCNN [45] | 67.7 | 79.5 | 73.4 | 84.1
SFNet [29] | 67.6 | 78.6 | 74.3 | 84.0
EMANet [31] | 65.6 | 77.7 | 72.9 | 83.1
RANet [41] | 66.1 | 78.2 | 73.8 | 83.9
EncodingNet [63] | 65.5 | 77.4 | 73.4 | 83.5
Denseaspp [53] | 64.7 | 76.4 | 73.9 | 83.9
PFNet | 70.4 | 81.9 | 75.4 | 84.8

Table 5: Comparison with the state-of-the-art results on Vahihigen(left) and
Potsdam(right) datasets.

###  4.2 Results on general segmentation benchmarks:

We further verify our approach on general segmentation benchmarks including
Cityscapes [12], ADE-20k [70] and BDD [56] for only verification purpose. We
only report the results due to the limited space. More implementation details
and visual results can be found in the supplementary file. We train both our
baseline model and PFnet model on train datasets and report results on
validation datasets under the same setting.

Method | Cityscapes | ADE20k | BDD | Param(M) | GFlops(G)
---|---|---|---|---|---
PSPNet [68] | 78.0 | 41.3 | 61.3 | 31.1 | 120.4
OCNet [59] | 79.2 | 41.8 | 62.1 | 64.7 | 290.4
Deeplabv3+ [9] | 79.4 | 42.0 | 61.0 | 40.5 | 189.8
baseline +PPM | 78.8 | 40.9 | 61.1 | 32.9 | 83.1
Our PFnet | 80.3 | 42.4 | 62.7 | 33.0 | 85.8

Table 6: Experiment results on general datasets including Cityscapes, ADE20k,
BDD validation datasets. All the methods are trained under the same training
setting and the results are reported with single scale inputs. The GFlops is
calculated with $512\times 512$ as input. All the methods use the ResNet50
backbone.

Comparison with the Baseline Methods: As shown in the last two rows of Tab. 6,
our method improves the baseline model on various datasets about 1% mIoU with
fewer parameters and GFlops increase. Compared with the previous work [68, 59,
9], our method achieves better results with much less computation cost.

##  5 Conclusion

In this paper, we propose PointFlow Module to solve both imbalanced
foreground-background objects and semantic gaps between feature pyramids
problems for aerial image segmentation. We design a novel Dual Point Matcher
to sampled the matched points from salient areas and boundaries accordingly.
Extensive experiments have shown that our PF module can improve various
baselines significantly on aerial benchmark. Based on the FPN framework, we
build an efficient PFNet which achieves the best speed and accuracy trade-off
on three public aerial benchmarks. Further experiments on three general
segmentation datasets also prove the generality of our method.

##  6 Acknowledgement

Much thanks to the SenseTime Research’s GPU servers for benchmarking semantic
segmentation results on these aerial datasets. Y. Tong is supported by the
National Key Research and Development Program of China (No.2020YFB2103402). Z.
Lin is supported by NSF China (grant no.s 61625301 and 61731018), Major
Scientific Research Project of Zhejiang Lab (grant no.s 2019KB0AC01 and
2019KB0AB02), Beijing Academy of Artificial Intelligence, and Qualcomm.

##  7 Supplementary

Overview: In this supplementary, we will present more experiments details on
Aerial datasets in the first section. Then we will provide detailed
descriptions and visualization results on general semantic segmentation
datasets.

###  7.1 Experiments on Aerial Datasets

In this section, we give supplementary aerial dataset experiments for the main
paper due to the space limitation.

Application on Various Methods: For Deeplabv3 [8], we append our PFNet decoder
after the ASPP output which makes the total network as an encoder-decorder
framework like U-net [43]. For CCnet [20], we follow the same pipeline of
Deeplabv3 by replacing ASPP head with CC-head. The GFlops in the table are
calculated with $512\times 512$ image inputs.

Effectiveness on Segmentation Boundaries: We present the our subtraction based
boundary prediction in Fig. 8. As shown in the figure, our subtraction based
prediction leads to thinner and sharpen prediction in the first row and second
row of Fig. 8 and avoids the inner part noise shown in the third row of Fig.
8.

Effect of Context Head: We also verify the effectiveness of context head by
replacing PPM [68] into ASPP [8] where we obtain 66.0 mIoU (0.9 mIoU drop
compared with PPM head). Due to the efficiency of PPM, we choose it as our
final context head.

Foreground Points Ratio of Propagation: We count the sampled points from the
three different PFMs by finding the intersection of all matched points from
Dual Point Matcher in each PFM. We match all the indexes into the high
resolution map during the calculation. Thus the propagated points with no-
overlap are considered.

More Visualization on iSAID dataset: Fig.7 provides more visualization results
on iSAID datasets. Compared with previous work, our method has better
segmentation results and less false positives in the scene. Fig. 9 shows more
matched points visualization on full cropped images.

Detailed Results: Tab. 7 reports the per-class results on iSAID datasets. From
that table, our PFNet achieves best results on 14 classes (total 15 classes).
Tab. 9 and Tab. 8 report per-class results on Postdam and Vaihingen dataset
receptively. Our PFNet also achieves top performance.

Dataset Split on Potsdam and Vaihingen: We provide the detailed dataset split
as following:

The Potsdam dataset consists of 38 high resolution aerial images. To train and
evaluate networks, we utilize 24 images for training and 14 images for
testing. For training set, the image IDs are 2_10, 2_11, 2_12, 3_10, 3_11,
3_12, 4_10, 4_11, 4_12, 5_10, 5_11, 5_12, 6_7, 6_8, 6_9, 6_10, 6_11, 6_12,
7_7, 7_8, 7_9, 7_10, 7_11, 7_12. For test set, the image IDs are 2_13, 2_14,
3_13, 3_14, 4_13, 4_14, 4_15, 5_13, 5_14, 5_15, 6_13, 6_14, 6_15, 7_13.

The Vaihingen dataset has 33 high resolution aerial images. We selects 16
images for trainand and another 17 images as the testset to evaluate the
models. The image IDs for training set are 1, 3, 5, 7, 11, 13, 15, 17, 21, 23,
26, 28, 30, 32, 34, 37 and image IDs for test set are 2, 4, 6, 8, 10, 12, 14,
16, 20, 22, 24, 27, 29, 31, 33, 35, 38.

Speed test details: We test our models on single V100 GPU by averaging
inference time with 100 image. The Pytorch version is 1.5 with CUDA-10.1.

Figure 7: Visualization results on iSAID validation dataset. Best view it on
screen. Figure 8: Visualization boundary results on iSAID validation dataset.
Best view it on screen. Figure 9: Visualization sampled points on iSAID
validation dataset. Best view it on screen.

Method | backbone | mIoU(%) | IoU per category(%) |
---|---|---|---|---
Ship | ST | BD | TC | BC | GTF | Bridge | LV | SV | HC | SP | RA | SBF | Plane | Harbor
DenseASPP [53] | RenNet50 | 57.3 | 55.7 | 63.5 | 67.2 | 81.7 | 54.8 | 52.6 | 34.7 | 55.6 | 36.3 | 33.4 | 37.5 | 53.4 | 73.3 | 74.7 | 46.7
RefineNet [34] | ResNet50 | 60.2 | 63.8 | 58.6 | 72.3 | 85.3 | 61.1 | 52.8 | 32.6 | 58.2 | 42.4 | 23.0 | 43.4 | 65.6 | 74.4 | 79.9 | 51.1
PSPNet [68] | ResNet50 | 60.3 | 65.2 | 52.1 | 75.7 | 85.6 | 61.1 | 60.2 | 32.5 | 58.0 | 43.0 | 10.9 | 46.8 | 68.6 | 71.9 | 79.5 | 54.3
OCNet-(ASP-OC) [59] | ResNet50 | 40.2 | 47.3 | 40.2 | 44.4 | 65.0 | 24.1 | 29.9 | 2.71 | 46.3 | 13.6 | 10.3 | 34.6 | 37.9 | 41.4 | 68.1 | 38.0
EMANet [31] | ResNet50 | 55.4 | 63.1 | 68.4 | 66.2 | 82.7 | 56.0 | 18.8 | 42.1 | 58.2 | 41.0 | 33.4 | 38.9 | 46.9 | 46.4 | 78.5 | 47.5
CCNet [20] | ResNet50 | 58.3 | 61.4 | 65.7 | 68.9 | 82.9 | 57.1 | 56.8 | 34.0 | 57.6 | 38.3 | 31.6 | 36.5 | 57.2 | 75.0 | 75.8 | 45.9
EncodingNet [63] | ResNet50 | 58.9 | 59.7 | 64.9 | 70.0 | 84.2 | 55.2 | 46.3 | 36.8 | 57.2 | 38.7 | 34.8 | 42.4 | 59.8 | 69.8 | 76.1 | 48.0
SemanticFPN [22] | ResNet50 | 62.1 | 68.9 | 62.0 | 72.1 | 85.4 | 54.1 | 48.9 | 44.9 | 61.0 | 48.6 | 37.4 | 42.8 | 70.2 | 58.6 | 84.7 | 54.9
UPerNet [22] | ResNet50 | 63.8 | 68.7 | 71.0 | 73.1 | 85.5 | 55.3 | 57.3 | 43.0 | 61.3 | 45.6 | 30.3 | 45.7 | 68.7 | 75.1 | 84.3 | 56.2
HRNet[51] | HRNetW18 | 61.5 | 65.9 | 68.9 | 74.0 | 86.9 | 59.4 | 61.5 | 33.8 | 62.1 | 46.9 | 14.9 | 44.2 | 52.9 | 75.6 | 81.7 | 52.2
SFNet[29] | ResNet50 | 64.3 | 68.8 | 71.3 | 72.1 | 85.6 | 58.8 | 60.9 | 43.1 | 62.9 | 47.7 | 30.4 | 47.8 | 69.8 | 75.1 | 83.1 | 57.3
GSCNN[45] | ResNe50 | 63.4 | 65.9 | 71.2 | 72.6 | 85.5 | 56.1 | 58.4 | 40.7 | 63.8 | 51.1 | 33.8 | 48.8 | 58.5 | 72.5 | 83.6 | 54.4
RANet[41] | ResNet50 | 62.1 | 67.1 | 61.3 | 72.5 | 85.1 | 53.2 | 47.1 | 45.3 | 60.1 | 49.3 | 38.1 | 41.8 | 70.5 | 58.8 | 83.1 | 55.6
FarSeg [69] | ResNet50 | 63.7 | 65.4 | 61.8 | 77.7 | 86.4 | 62.1 | 56.7 | 36.7 | 60.6 | 46.3 | 35.8 | 51.2 | 71.4 | 72.5 | 82.0 | 53.9
PFNet | ResNet50 | 66.9 | 70.3 | 74.7 | 77.8 | 87.7 | 62.2 | 59.5 | 45.2 | 64.6 | 50.2 | 37.9 | 50.1 | 71.7 | 75.4 | 85.0 | 59.3

Table 7: Experimental results on iSAID $val$ set. The bold values in each
column mean the best entries. The category are defined as: ship (Ship),
storage tank (ST), baseball court (BC), ground field track (GTF), bridge
(Bridge), large vehicle (LV), small vehicle (SV), helicopter (HC), swimming
pool (SP), roundabout (RA), soccerball field (SBF), plane (Plane), harbor
(Harbor). All the models are trained under the same setting following the
FarSeg [69].

Method | mIoU(%) | mean-$F_{1}$ |  $F_{1}$ per category |
---|---|---|---|---
Imp.surf. | Build. | Low veg. | Tree | Car | Cluster
PSPNet [68] | 65.1 | 76.8 | 88.4 | 92.8 | 79.2 | 85.9 | 73.5 | 41.0
FCN [37] | 64.2 | 75.9 | 87.6 | 91.6 | 77.8 | 84.6 | 73.5 | 40.3
OCNet(ASP-OC) [59] | 65.7 | 77.4 | 88.8 | 92.9 | 79.2 | 85.8 | 73.9 | 43.8
Deeplabv3+ [9] | 64.3 | 76.0 | 88.7 | 92.8 | 78.9 | 85.6 | 72.4 | 37.6
DANet [16] | 65.3 | 77.1 | 88.5 | 92.7 | 78.8 | 85.7 | 73.7 | 43.2
CCNet [20] | 64.3 | 75.9 | 88.3 | 92.5 | 78.8 | 85.7 | 73.9 | 36.3
SemanticFPN [22] | 66.3 | 77.6 | 89.6 | 93.6 | 79.7 | 86.3 | 75.7 | 40.7
UPerNet [51] | 66.9 | 78.7 | 89.2 | 93.0 | 79.4 | 86.0 | 74.9 | 49.7
PointRend [23] | 65.9 | 78.1 | 88.2 | 92.4 | 78.9 | 84.5 | 73.5 | 51.1
HRNet-W18 [47] | 66.9 | 78.2 | 89.2 | 92.6 | 78.7 | 85.7 | 77.1 | 45.9
GSCNN [45] | 67.7 | 79.5 | 89.4 | 92.6 | 78.8 | 85.4 | 77.9 | 52.9
SFNet [29] | 67.6 | 78.6 | 90.0 | 94.0 | 80.3 | 86.5 | 78.9 | 41.9
EMANet [31] | 65.6 | 77.7 | 88.2 | 92.7 | 78.0 | 85.7 | 72.7 | 48.9
RANet [41] | 66.1 | 78.2 | 88.0 | 92.3 | 79.1 | 86.0 | 78.8 | 53.1
EncodingNet [63] | 65.5 | 77.4 | 88.6 | 92.5 | 78.5 | 85.7 | 73.6 | 45.5
Denseaspp [53] | 64.7 | 76.4 | 87.3 | 91.1 | 76.2 | 83.4 | 77.1 | 43.3
PFNet | 70.4 | 81.9 | 90.1 | 93.6 | 77.7 | 85.4 | 80.0 | 64.6

Table 8: Experimental results on the Vaihingen Dataset. The results are
reported with single scale input. The bold values in each column mean the best
entries. The category are defined as: impervious surfaces (Imp.surf.),
buildings (Build), low vegetation (Low veg), trees (Tree), cars (Car),
cluster/background (Cluster).

Method | mIoU(%) | mean-$F_{1}$ |  $F_{1}$ per category |
---|---|---|---|---
Imp.surf. | Build. | Low veg. | Tree | Car | Cluster
PSPNet [68] | 73.9 | 83.9 | 90.8 | 95.4 | 84.5 | 86.1 | 88.6 | 58.0
FCN [37] | 73.1 | 83.1 | 90.2 | 94.7 | 84.1 | 85.6 | 89.2 | 54.8
OCnet(ASP-OC) [59] | 74.2 | 84.1 | 90.9 | 95.5 | 84.8 | 86.0 | 89.2 | 58.2
Deeplabv3+ [9] | 74.1 | 83.9 | 91.0 | 95.6 | 84.6 | 86.0 | 90.0 | 56.2
DAnet [16] | 74.0 | 83.9 | 91.0 | 95.6 | 84.9 | 86.2 | 88.7 | 57.0
CCnet[20] | 73.8 | 83.8 | 90.7 | 95.5 | 84.7 | 86.0 | 88.5 | 57.3
SemanticFPN [22] | 74.3 | 84.0 | 91.0 | 95.5 | 84.9 | 85.9 | 90.4 | 56.3
UPerNet [51] | 74.3 | 84.0 | 90.9 | 95.7 | 85.0 | 86.0 | 90.2 | 56.2
PointRend [23] | 72.0 | 82.7 | 89.8 | 94.6 | 82.8 | 85.2 | 85.2 | 58.6
HRNet-W18 [47] | 73.4 | 83.4 | 90.4 | 94.9 | 84.2 | 85.4 | 90.0 | 55.5
GSCNN [45] | 73.4 | 84.1 | 91.4 | 95.5 | 84.8 | 85.8 | 91.2 | 55.9
SFNet [29] | 74.3 | 84.0 | 91.0 | 95.5 | 85.1 | 86.0 | 90.9 | 55.5
EMANet [31] | 72.9 | 83.1 | 90.4 | 94.9 | 84.2 | 85.7 | 88.3 | 55.1
RANet [41] | 73.8 | 83.9 | 90.8 | 92.1 | 84.3 | 86.8 | 88.9 | 56.0
EncodingNet [63] | 73.4 | 83.5 | 90.6 | 95.1 | 84.5 | 86.0 | 88.2 | 56.6
Denseaspp [53] | 73.9 | 83.9 | 90.8 | 95.4 | 84.6 | 86.0 | 88.5 | 58.1
PFNet | 75.4 | 84.8 | 91.5 | 95.9 | 85.4 | 86.3 | 91.1 | 58.6

Table 9: Experimental results on Postdam Dataset. The results are reported
with single scale input. The bold values in each column mean the best entries.
The category are defined as: impervious surfaces (Imp.surf.), buildings
(Build), low vegetation (Low veg), trees (Tree), cars (Car),
cluster/background (Cluster).

###  7.2 Experiments on Generation Segmentation Dataset

In this section, we will first give the implementation details on Cityscapes
[12], BDD [56] and ADE20k [70].

Implementation Details

Cityscapes dataset is composed of a large set of high-resolution $(2048\times
1024)$ images in street scenes. This dataset has 5,000 images with high
quality pixel-wise annotations for 19 classes, which is further divided into
2975, 500, and 1525 images for training, validation and testing. We only use
the fine-data for training. Following the common practice, the “poly” learning
rate policy is adopted to decay the initial learning rate by multiplying
$(1-\frac{\text{iter}}{\text{total}\\_\text{iter}})^{0.9}$ during training.
Data augmentation contains random horizontal flip, random resizing with scale
range of $[0.75,~{}2.0]$, and random cropping with crop size of $864\times
864$ and we train totally 300 epochs for strong baseline.

BDD is a new road scene benchmark consisting $7,000$ images for training and
$1,000$ images for validation. We follow the same setting as Cityscapes
dataset.

ADE20k is a more challenging scene parsing dataset annotated with 150 classes,
and it contains 20K/2K images for training and validation. It has the various
objects in the scene. We train the network for 120 epochs with batch size 16,
crop size 512 and initial learning rate 1e-2. For final testing, we perform
multi-scale testing with horizontal flip operation.

Visualization and Comparison Results We provide visualization and comparison
results on Cityscapes and ADE-20k in Fig. 10 and Fig. 11. From both figures,
our PFM can better handle the segmentation tails and missing objects in the
scene.

Figure 10: Visualization results on Cityscapes dataset. Best view it on
screen.  Figure 11: Visualization results on ADE20k dataset. Best view it on
screen.

## References

  * [1] E. H. Adelson, C. H. Anderson, J. R. Bergen, P. J. Burt, and J. M. Ogden.  Pyramid methods in image processing.  RCA engineer, 29(6), 1984.
  * [2] V. Badrinarayanan, A. Kendall, and R. Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  PAMI, 2017.
  * [3] F. Bastani, S. He, S. Abbar, M. Alizadeh, H. Balakrishnan, S. Chawla, S. Madden, and D. DeWitt.  Roadtracer: Automatic extraction of road networks from aerial images.  In CVPR, pages 4720–4728, 2018.
  * [4] P. Bilinski and V. Prisacariu.  Dense decoder shortcut connections for single-pass semantic segmentation.  In CVPR, 2018.
  * [5] P. J. Burt.  Fast filter transform for image processing.  Computer graphics and image processing, 16(1):20–51, 1981.
  * [6] H. Caesar, J. Uijlings, and V. Ferrari.  Coco-stuff: Thing and stuff classes in context.  In CVPR, 2018.
  * [7] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  PAMI, 2018.
  * [8] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint, 2017.
  * [9] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In ECCV, 2018.
  * [10] Y. Chen, M. Rohrbach, Z. Yan, Y. Shuicheng, J. Feng, and Y. Kalantidis.  Graph-based global reasoning networks.  In CVPR, 2019.
  * [11] M. T. Chiu, X. Xu, Y. Wei, Z. Huang, A. G. Schwing, R. Brunner, H. Khachatrian, H. Karapetyan, I. Dozier, G. Rose, et al.  Agriculture-vision: A large aerial image database for agricultural pattern analysis.  In CVPR, 2020.
  * [12] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, 2016.
  * [13] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei.  Deformable convolutional networks.  In ICCV, 2017.
  * [14] M. Dickenson and L. Gueguen.  Rotated rectangles for symbolized building footprint extraction.  In CVPR Workshops, pages 225–228, 2018.
  * [15] H. Ding, X. Jiang, B. Shuai, A. Qun Liu, and G. Wang.  Context contrasted feature and gated multi-scale aggregation for scene segmentation.  In CVPR, 2018.
  * [16] J. Fu, J. Liu, H. Tian, Z. Fang, and H. Lu.  Dual attention network for scene segmentation.  In CVPR, 2019.
  * [17] G. Ghiasi, T.-Y. Lin, and Q. V. Le.  Nas-fpn: Learning scalable feature pyramid architecture for object detection.  In CVPR, 2019.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [19] B. Huang, B. Zhao, and Y. Song.  Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery.  Remote Sensing of Environment, 214:73–86, 2018.
  * [20] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu.  Ccnet: Criss-cross attention for semantic segmentation.  In ICCV, 2019.
  * [21] P. Kaiser, J. D. Wegner, A. Lucchi, M. Jaggi, T. Hofmann, and K. Schindler.  Learning aerial image segmentation from online maps.  IEEE Transactions on Geoscience and Remote Sensing, 55(11):6054–6068, 2017.
  * [22] A. Kirillov, R. Girshick, K. He, and P. Dollár.  Panoptic feature pyramid networks.  In CVPR, pages 6399–6408, 2019.
  * [23] A. Kirillov, Y. Wu, K. He, and R. Girshick.  PointRend: Image segmentation as rendering.  In CVPR, 2020.
  * [24] D. Li, J. Hu, C. Wang, X. Li, Q. She, L. Zhu, T. Zhang, and Q. Chen.  Involution: Inverting the inherence of convolution for visual recognition.  In CVPR, June 2021.
  * [25] X. Li, Z. Houlong, H. Lei, T. Yunhai, and Y. Kuiyuan.  Gff: Gated fully fusion for semantic segmentation.  In AAAI, 2020.
  * [26] X. Li, X. Li, A. You, L. Zhang, G.-L. Cheng, K. Yang, Y. Tong, and Z. Lin.  Towards efficient scene understanding via squeeze reasoning.  ArXiv, abs/2011.03308, 2020.
  * [27] X. Li, X. Li, L. Zhang, G. Cheng, J. Shi, Z. Lin, S. Tan, and Y. Tong.  Improving semantic segmentation via decoupled body and edge supervision.  In ECCV, 2020.
  * [28] X. Li, Y. Yang, Q. Zhao, T. Shen, Z. Lin, and H. Liu.  Spatial pyramid based graph reasoning for semantic segmentation.  In CVPR, pages 8950–8959, 2020.
  * [29] X. Li, A. You, Z. Zhu, H. Zhao, M. Yang, K. Yang, and Y. Tong.  Semantic flow for fast and accurate scene parsing.  ECCV, 2020.
  * [30] X. Li, L. Zhang, A. You, M. Yang, K. Yang, and Y. Tong.  Global aggregation then local distribution in fully convolutional networks.  arXiv preprint arXiv:1909.07229, 2019.
  * [31] X. Li, Z. Zhong, J. Wu, Y. Yang, Z. Lin, and H. Liu.  Expectation-maximization attention networks for semantic segmentation.  In ICCV, 2019.
  * [32] Y. Li and A. Gupta.  Beyond grids: Learning graph representations for visual recognition.  In NeurIPS. 2018.
  * [33] J. Liang, N. Homayounfar, W.-C. Ma, S. Wang, and R. Urtasun.  Convolutional recurrent network for road boundary extraction.  In CVPR, pages 9512–9521, 2019.
  * [34] G. Lin, A. Milan, C. Shen, and I. D. Reid.  Refinenet: Multi-path refinement networks for high-resolution semantic segmentation.  In CVPR, 2017.
  * [35] T.-Y. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie.  Feature pyramid networks for object detection.  In CVPR, 2017.
  * [36] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia.  Path aggregation network for instance segmentation.  In CVPR, pages 8759–8768, 2018.
  * [37] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In CVPR, 2015.
  * [38] D. Marcos, M. Volpi, B. Kellenberger, and D. Tuia.  Land cover mapping at very high resolution with rotation equivariant cnns: Towards small yet accurate models.  ISPRS journal of photogrammetry and remote sensing, 145:96–107, 2018.
  * [39] D. Marmanis, K. Schindler, J. D. Wegner, S. Galliani, M. Datcu, and U. Stilla.  Classification with an edge: Improving semantic image segmentation with boundary detection.  ISPRS Journal of Photogrammetry and Remote Sensing, 135:158–172, 2018.
  * [40] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille.  The role of context for object detection and semantic segmentation in the wild.  In CVPR, 2014.
  * [41] L. Mou, Y. Hua, and X. X. Zhu.  A relation-augmented fully convolutional network for semantic segmentation in aerial scenes.  In CVPR, pages 12416–12425, 2019.
  * [42] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In CVPR, 2016.
  * [43] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  MICCAI, 2015.
  * [44] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  ICLR, 2015.
  * [45] T. Takikawa, D. Acuna, V. Jampani, and S. Fidler.  Gated-scnn: Gated shape cnns for semantic segmentation.  ICCV, 2019.
  * [46] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [47] J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang, W. Liu, and B. Xiao.  Deep high-resolution representation learning for visual recognition.  TPAMI, 2019.
  * [48] X. Wang, R. Girshick, A. Gupta, and K. He.  Non-local neural networks.  In CVPR, 2018.
  * [49] X. Wang, R. Girshick, A. Gupta, and K. He.  Non-local neural networks.  In CVPR, 2018.
  * [50] S. Waqas Zamir, A. Arora, A. Gupta, S. Khan, G. Sun, F. Shahbaz Khan, F. Zhu, L. Shao, G.-S. Xia, and X. Bai.  isaid: A large-scale dataset for instance segmentation in aerial images.  In ICCV Workshops, 2019.
  * [51] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun.  Unified perceptual parsing for scene understanding.  In ECCV, 2018.
  * [52] Y. Xu, L. Wu, Z. Xie, and Z. Chen.  Building extraction in very high resolution remote sensing imagery using deep learning and guided filters.  Remote Sensing, 10(1):144, 2018.
  * [53] M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang.  Denseaspp for semantic segmentation in street scenes.  In CVPR, 2018.
  * [54] C. Yu, Y. Liu, C. Gao, C. Shen, and N. Sang.  Representative graph neural network.  In ECCV, pages 379–396. Springer, 2020.
  * [55] C. Yu, J. Wang, C. Gao, G. Yu, C. Shen, and N. Sang.  Context prior for scene segmentation.  In CVPR, pages 12416–12425, 2020.
  * [56] F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In CVPR, 2020.
  * [57] F. Yu and V. Koltun.  Multi-scale context aggregation by dilated convolutions.  ICLR, 2016.
  * [58] Y. Yuan, X. Chen, and J. Wang.  Object-contextual representations for semantic segmentation.  ECCV, 2020.
  * [59] Y. Yuan and J. Wang.  Ocnet: Object context network for scene parsing.  arXiv preprint, 2018.
  * [60] Y. Yuan, J. Xie, X. Chen, and J. Wang.  Segfix: Model-agnostic boundary refinement for segmentation.  In ECCV, 2020.
  * [61] D. Zhang, H. Zhang, J. Tang, M. Wang, X. Hua, and Q. Sun.  Feature pyramid transformer.  ECCV, 2020.
  * [62] F. Zhang, Y. Chen, Z. Li, Z. Hong, J. Liu, F. Ma, J. Han, and E. Ding.  Acfnet: Attentional class feature network for semantic segmentation.  In ICCV, 2019.
  * [63] H. Zhang, K. Dana, J. Shi, Z. Zhang, X. Wang, A. Tyagi, and A. Agrawal.  Context encoding for semantic segmentation.  In CVPR, 2018.
  * [64] H. Zhang, H. Zhang, C. Wang, and J. Xie.  Co-occurrent features in semantic segmentation.  In CVPR, 2019.
  * [65] L. Zhang, X. Li, A. Arnab, K. Yang, Y. Tong, and P. H. Torr.  Dual graph convolutional network for semantic segmentation.  In BMVC, 2019.
  * [66] L. Zhang, D. Xu, A. Arnab, and P. H. Torr.  Dynamic graph message passing networks.  In CVPR, June 2020.
  * [67] Z. Zhang, X. Zhang, C. Peng, X. Xue, and J. Sun.  Exfuse: Enhancing feature fusion for semantic segmentation.  In ECCV, 2018.
  * [68] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.  Pyramid scene parsing network.  In CVPR, 2017.
  * [69] Z. Zheng, Y. Zhong, J. Wang, and A. Ma.  Foreground-aware relation network for geospatial object segmentation in high spatial resolution remote sensing imagery.  In CVPR, pages 4096–4105, 2020.
  * [70] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba.  Semantic understanding of scenes through the ADE20K dataset.  arXiv preprint, 2016.
  * [71] X. Zhu, H. Hu, S. Lin, and J. Dai.  Deformable convnets v2: More deformable, better results.  In CVPR, 2019.
  * [72] Z. Zhu, M. Xu, S. Bai, T. Huang, and X. Bai.  Asymmetric non-local neural networks for semantic segmentation.  In ICCV, 2019.