This research was partially financed by the Coordenação de Aperfeiçoamento de
Pessoal de Nível Superior (CAPES), Fundação de Amparo à Pesquisa do Estado de
Minas Gerais (FAPEMIG), Fundação de Amparo à Pesquisa do Estado de São Paulo
(FAPESP – grant #2020/06744-5), Agence Nationale de la Recherche (ANR) and the
Serrapilheira Institute (grant R-2011-37776). The authors would also like to
thank NVIDIA for the donation of the GPUs used in this research.

Corresponding author: Ian Nunes (e-mail: inunes@inf.puc-rio.br).

# FuSS: Fusing Superpixels for Improved Segmentation Consistency

IAN NUNES1  MATHEUS B. PEREIRA2  HUGO OLIVEIRA3  JEFERSSON A. DOS SANTOS2 and
MARCUS POGGI1  Department of Informatics, at Pontifical Catholic University of
Rio de Janeiro, Rio de Janeiro, Brazil (e-mail<EMAIL_ADDRESS>Department of Computer Science (DCC) in Universidade Federal de Minas Gerais
(UFMG), Belo Horizonte, Brazil (e-mail<EMAIL_ADDRESS> Institute
of Mathematics and Statistics (IME), at University of São Paulo (USP), São
Paulo, Brazil (e-mail<EMAIL_ADDRESS>
###### Abstract

In this work, we propose two different approaches to improve the semantic
consistency of Open Set Semantic Segmentation. First, we propose a method
called OpenGMM that extends the OpenPCS framework using a Gaussian Mixture of
Models to model the distribution of pixels for each class in a multimodal
manner. The second approach is a post-processing which uses superpixels to
enforce highly homogeneous regions to behave equally, rectifying erroneous
classified pixels within these regions, we also proposed a novel superpixel
method called FuSS. All tests were performed on ISPRS Vaihingen and Potsdam
datasets, and both methods were capable to improve quantitative and
qualitative results for both datasets. Besides that, the post-process with
FuSS achieved state-of-the-art results for both datasets. The official
implementation is available at: https://github.com/iannunes/FuSS.

###### Index Terms:

convolutional networks, open set semantic segmentation, remote sensing,
semantic consistency, superpixel segmentation

=-15pt

##  I Introduction

Remote sensing acquisition technologies have been in constant development
since the 1960s, providing sensors with a myriad of new electromagnetic
spectral encoding capabilities and leading to a continuous increase in the
volume of daily collected data. The development of these technologies,
together with an increased capacity to produce relevant information for a wide
range of applications, turned the automatic analysis of images into one of the
most actively researched fields within the remote sensing community [1].

Since the Gestalt movement, it is known that image segmentation and clustering
plays an important role in human perception [2]. Many different applications
can benefit from semantic segmentation of remote sensing images, such as:
urban planning/mapping and change detection [3], population estimation [4],
real-estate management [5].

Figure 1: Post-processing procedure for OSS proposed in this work. First the
input image is processed by the chosen open set segmentation method.
Afterwards, the likelihood scores generated by the OSS method (e.g.
reconstruction errors, likelihood scores, etc) are processed using superpixel
segmentation and the mean score of each segment score is set to the whole
area. The final step includes thresholding the predictions according to some
cutoff – usually set according to quantiles on the scores – to identify
unknown superpixels.

Traditional (closed set) semantic segmentation (SSeg) for remote sensing
imagery has become a complex, time-consuming, and well-studied task. Many
different methods have been developed to solve this problem: FCN [6], U-Net
[7], SegNet [8], among others. In recent years, many models have significantly
improved the quality of semantic segmentation [9, 10].

To enable the use of SSeg models in practical situations, it is crucial to
improve the semantic and spatial consistency of the resultant segmentation.
According to Sekkal et al. [11], it is important for the regions to remain
coherent with the original content and the better detection of contours leads
to an efficient pseudo-semantic representation.

In closed-set SSeg, both the training and test data share the same label and
feature spaces. However, in more realistic scenarios unseen classes may show
up on the deploy phase of the model. This is the case for most real-world
applications, such as autonomous vehicles, medical diagnosis or treatment, and
remote sensing tasks. The existence of unknown classes undermines the
robustness of the existing closed set methods, as stated by Geng et al. [12].

For the last couple of years, methods that extend traditional closed set SSeg
were proposed to automatically recognize samples from unseen classes. This new
task, named Open Set Segmentation (OSS), must be able to correctly segment
pixels of classes available during training (Known Known Classes – KKCs),
while also being able to recognize unknown pixels that come from classes that
were not present during training (Unknown Unknown Classes – UUCs) [13].

The goal of the current paper is to improve the semantic consistency of
previously obtained results OSS algorithms for remote sensing images. In order
to improve segmentation consistency, we introduce in this work a novel method
called OpenGMM based on previously proposed frameworks for OSR [13, 14].
Oliveira et al. [13] employed Principal Component Analysis (PCA) in the
OpenPCS method as a generative model, assuming it would be sufficient for the
representation of the data. As data in the real world can rarely be
represented by unimodal distributions, our proposed approach adopts a Gaussian
Mixture Model (GMM) [15, 16] instead of PCA, allowing the use of multimodal
distributions for modeling KKCs with the objective of improving the out-of-
distribution (OOD) identification.

The lack of semantic consistency produced by OpenPCS, specially in objects
borders, can be seen in Figure 1. Threshold methods like OpenPCS presents
difficulties in correctly delimiting the edges between objects making these
kind of OSS approaches not likely to be used in real world scenarios. To
address this issue and improve the final results we studied ways to add a
post-process using Superpixel Segmentation (SPS) with qualitative results
shown in Figure 1, the post processed OSS is closer to the ground truth and
many generated artifacts were removed.

Superpixels are a oversegmentation of an image and played an important role in
the process of traditional segmentation, and acoording to Lin et al. [17] are
a low-level image information subdivisions. As stated by [18, 19, 20] the use
of superpixels: allows for a richer feature representation for each segment
(textural, color, scatter, gradient-based, statistic, orientation); reduces
the size of the classification problem since many pixels could be represented
by each cluster; even being an oversegmentation of the image the superpixel
clustering produces homogeneous regions that adds semantics to each
superpixel. Superpixels are a homogeneous and contiguous group of pixels in an
image extracting perceptually relevant regions [18]. The segmentation
generated with superpixels reduces the input size, while preserving the
semantic content needed to address intended tasks [19].

Our framework employs a novel superpixel merge procedure using Malahanobis
distance [21], called Fusing Superpixels for Segmentation (FuSS). A graphical
illustration of our pipeline’s inner workings can be seen in Figure 1. We
tested our approach on OSS using three distinct algorithms: OpenPCS [13],
CoReSeg [22] and the newly proposed OpenGMM.

Our pipeline seeks to improve the semantic consistency of the results in order
to make OSS models more suitable for practical use. Our contributions can be
summarized as:

  * •

The proposal of OpenGMM, a novel method aiming to improve upon the previously
proposed OSS framework by Oliveira et al. [13];

  * •

The proposal of a superpixel post-processing method that yielded results with
superior semantic consistency and improved Receiver Operating Characteristic
metrics in all tested scenarios;

  * •

A novel superpixel segmentation fusion procedure using Malahanobis distance
[21];

  * •

State of the Art OSS results for the Vaihingen and Potsdam
datasets111https://www2.isprs.org/commissions/comm2/wg4/benchmark/ using our
post-processing segmentation technique;

This manucript is organized as follows: Section II presents previous OSS
strategies and related work on superpixels for semantic segmentation; Section
III describes the proposed methods; Section IV introduces the experimental
setup used for this work, along with the employed datasets and metrics;
Section V presents exploratory tests executed on Vaihingen dataset; Section VI
presents the obtained results; and Section VII exposes the conclusions over
the proposed methods and the achieved results.

##  II Related Work

###  II-A Semantic Consistency in Segmentation

Semantic consistency is rarely explicitly addressed in semantic segmentation
papers. In the following lines we present an overview of the few existing
trends on deep semantic consistency.

Through an end-to-end trainable network that combines 2 branches, one for edge
detection and one for traditional semantic segmentation, Ji et al. [23]
managed to improve the performance and the spatial consistency of the
resulting segmentation for PASCAL VOC 2012, PASCAL-Context and Cityscapes
datasets.

PixMatch, proposed by Melas-Kyriazi et al. [24], uses heavy augmentation and a
loss term composed by the summation of two cross-entropy terms. The first loss
term is standard for SSeg, and the second is calculated over a slightly
perturbed image and mask. The new loss enforces the notion of smoothness in
the target domain in order to enhance intra-object segmentation consistency.

Pixelwise Contrast and Consistency Learning (PiCoCo), proposed by Kang et al.
[25], seeks consistency in closed set semantic segmentation using a joint loss
function that is summation of a supervised loss term, a contrast loss term,
and a consistency loss term. The supervised is a standard semantic
segmentation loss term composed by a Cross Entropy and a Dice loss term; for
the contrastive loss a selection of positive and negative samples enforce the
model to improve its generalization capabilities; the consistency loss term
consists in a summation of a cross entropy and a dice loss of heavy augmented
pairs of input and labels to enforce semantic consistency and robustness to
the learning process.

The use of a post-processing that combines unsupervised colorization and deep
edge superpixels to enhance the semantic segmentation of panchromatic aerial
images was proposed by Ratajczak et al. [26]. The authors propose to assess if
applying a colorization algorithm could improve the strenght of the pairwise
potentials used in a conditional random field (CRF) post-processing. In this
work Deep Edge Superpixels (DES) are defined using the Watershed [27] with the
intermediate activation maps obtained before each pooling layer to the output
space of a Holistically-Nested Edge Detection Nework [28], they use the
generated SPS with mean value for intensity together with CRF to improve the
final semantic consistency.

Also the use of supervoxels to improve segmentation consistency was used by
Zhang et al. [29], where a 3DCNN was used to learn discriminative hierarchical
features from spatio-temporal volumes.

Our work introduced a post-processing for OSS that uses a SPS to improve the
semantic consistency of the resultant OSS, this procedure produced better
results in all tested scenarios. We also proposed a novel SPS called FuSS that
bennefits from the merge of different input segmentations and produces a SPS
with better results in most tested scenarios when compared to the same post-
processing with the single algorithm SPS.

###  II-B Open Set Semantic Segmentation

As stated by Scheirer et al. [30] an open set scenario happens when unknown
samples can appear in the prediction phase, meaning that at training time not
all possible classes are known. This definition can be applied for each pixel
in an image, extending the traditional semantic segmentation to OSS.

OSS has only a handful of published works that use neural networks. The first
attempts to do open set recognition from a neural network were proposed by
Bendale and Boult [31] the first natural approach was to apply a threshold on
the final output probability, identifying low probabilities as unknown. The
experiments showed that this strategy was weak and not enough to deal the
task. The second approach was a method called OpenMax, which presented a new
model layer to replace the traditional softmax. Assuming $C$ classes trained
on the closed set task, OpenMax adds an Unknown output class and estimates the
probability of the input images to each of the $C+1$ classes.

Based on OpenMax [31], OpenPixel was proposed by Silva et al. [32]. It uses a
patch-wise strategy to classify the central pixel. In practice, each patch
only classifies the central pixel, thus the network needs one patch per pixel
to segment the whole image. OpenPixel is, therefore, extremely inefficient
mainly during the test phase, as each pixel in an image generates a patch to
be fed to the network, with surrounding pixels being forwarded multiple times
through the backbone. The fully convolutional counterpart to OpenPixel, named
OpenFCN, was proposed by Oliveira et al. [13]. OpenMax-based methods proved to
have their efficacy severely limited in segmentation settings, resulting in
false positive OOD pixel predictions mainly in the borders of objects, where
the activations of the last layers are considerably affected by the presence
of surrounding objects with different classes.

Given the limitations of OpenPixel and OpenFCN caused by last layers’
activations, Oliveira et al. [13] proposed the use of intermediate multiscale
features from the closed set FCNs paired with low-dimensional principal
component scoring for OSS. The method, named Open Principal Component Scoring
(OpenPCS), achieved consistently better results than the OpenMax-based
approaches in multiple scenarios and unknown class configurations on the
Vaihingen, Potsdam and GRSS 2018 Data Fusion Challenge [33] datasets.

Chamorro et al. [34] built upon OpenPCS, including a whitening operation to
better generalize to a wide range of imbalanced classes. OpenPCS++ was tested
on multi-temporal SAR images from the LEM222http://www.lvc.ele.puc-
rio.br/LEM_benchmark/ and CV333https://ieee-dataport.org/documents/campo-
verde-database datasets in the task of segmenting OOD crops among known crop
species.

Cui et al. [35] proposed a nonparametric statistical OSS method that employs
the Mann-Whitney U test on a closed set segmentation output to determine the
existence of unknown classes in each image. Furthermore, it uses an adaptive
threshold that identify which pixels are unknown.

Nunes et al. [22] proposed a fully convolutional end-to-end Conditional
Reconstruction Open Set Segmentation (CoReSeg) that tackles the OSS using two
network branches: a traditional closed set segmentation branch and a class
conditional reconstruction of the input images according to their pixelwise
mask, together the branches merges into a single open set segmentation. The
traditional segmentation branch produces a standard closed set segmentation
and the reconstruction error from the conditional reconstruction branch
identifies which pixels are OOD.

Proposed by Cen et al. [36] an open world semantic segmentation system used
prototypes for the known classes and a Deep Metric Learning Network (DML-Net)
as a feature extractor. The final classifier is replaced by a representation
of the Euclidean distance of the input image to all prototypes. Low
probability classes are set to unknown with the use of a Euclidean distance-
based probability loss.

The Generalized Open-set Semantic Segmentation (GOSS) method was proposed by
Hong et al. [37]. The authors were inspired by the human capacity of
recognizing the previously known classes, while simultaneously identifying and
grouping the unknown objects into different categories, even without previous
knowledge about them. The method employs two network branches trained together
in parallel: the first branch performs a SSeg for KKCs and identifies unknown
pixels using Deep Metric Learning (DML) [36]the second branch is a pixel
clustering one that ignores the KKCs and generates a new segmentation mask to
the image. At last, the fusion phase uses the pixels defined as unknown and
the clustering to identify different objects in the unknown areas.

###  II-C Superpixel Segmentation (SPS) Algorithms

In this work, we consider a superpixel as a group of contiguous pixels in a
given image which have been grouped together according to some criterion of
homogeneity. As spatiality is crucial to any SPS, neighboring superpixels
should be perceptually different. Nevertheless, non-neighboring superpixels
may have similar values and shapes. All pixels inside a superpixel should be
represented by the mean or median value for each image band.

SPSs are an active research area, and many different methods have been
proposed to generate superpixels from an image. As examples of relevant
methods proposed in the last 2 decades, we can cite: Felzenszwalb [18],
Quickshift [38], TurboPixels [39], ERS [40], SLIC [41], GSM [42], Eikonal-
based [43], SEEDS [44], LSC [45], Waterpixels [46], BASS [47], SAS [48],
SH+FDAG [20], content-based [49] and SPFCM [50].

Among all possible choices of SPS algorithms to use in this work we choose
three algorithms that have fundamentally different strategies to generate the
superpixels: SLIC [41], Quickshift [38] and Felzenszwalb [18]. In the
following paragraphs we briefly present these three superpixel algorithms.

####  II-C1 Simple Linear Iterative Clustering

The SLIC algorithm was proposed by Achanta et al. [41] and its goal is to
group pixels into perceptually meaningful contiguous regions. The method
adapts the K-means algorithm [51] to generate the superpixels.

The operation of the algorithm is simple. It starts with $n$ centers uniformly
distributed in the image. Then, it adjusts the centers’ positions to the local
minimum of the gradient of pixel intensity as to avoid centering the
superpixel in an edge. Finally, it iterates to every center, reallocating the
pixels in a fixed size window to the closest center. Limiting the search area
makes the algorithm faster while still producing good results. The SLIC
algorithm produces areas with great homogeneity as it is built on a K-means
like approach, but it may neglect the natural edges of the image as a side
effect.

####  II-C2 Quickshift

Quickshift is a fast mode seeking algorithm proposed by Vedaldi et al. [38].
In Quickshift, every pixel is started as a superpixel, then the closest ones
are put together within a defined radius distance. For each superpixel, if a
pixel is out of the radius limits, a new cluster is defined and populated by
the pixels inside the radius of the new centroid. Quickshift is a hierarchical
clustering algorithm, the size of the clusters can be derived from the
generated tree of radius values. This method do not force the pixels to be
close to each other spatially, what generates highly homogeneous pixels of
different sizes and shapes.

####  II-C3 Felzenszwalb

Proposed by Felzenszwalb et al. [18], the algorithm is a graph based
segmentation algorithm where each vertex represents a pixel and each selected
edge has some dissimilarity measure as its value. Every pixel in the image is
represented in the graph, but only some of the edges are selected according to
a defined criterion (e.g. $K$-nearest neighbors) in order to guarantee the
intended complexity for the algorithm ($O(m\ log\ n)$, where $m$ is the number
of edges and $n$ the number of vertices).

There are two presented strategies to select the edges, the first takes into
account the notion of grid and connects each pixel to the 8 closest ones in
the grid. The second strategy maps the image into a higher level feature space
and connects the $m$ closest points in this new space. For instance, a
$5$-dimensional space defined by the spatial position $x$ and $y$, and the
three colors of RGB.

The algorithm explores the same idea that Kruskal et al. [52] used in their
classical algorithm to find the minimum spanning tree (MST) on a graph and
selects the edges in a non-descending order to generate the clusters. The
clusters are generated using the intuition that they are defined by a region
where the values of the edges are low and the borders of the clusters have
greater dissimilarities when compared to the intra-cluster values.

By construction, the Felzenszwalb algorithm generates clusters that vary in
shapes and sizes, but strongly respects the borders of the natural objects in
the image.

##  III Improving OSS Semantic Consistency

In the following section we present our proposed methodology for improving the
semantic consistency of OSS. Firstly, in Section III-A we present an extension
of the OSS framework proposed by Oliveira et al. [13], replacing the unimodal
dimensionality reduction of Principal Component Analysis (PCA) with a
multimodal Gaussian Mixture Model (GMM) capable of opening closed set
pretrained segmentation networks with a better segmentation quality. In
Section III-B we propose a superpixel post-processing for generic OSS methods
capable of improving Receiver Operating Characteristic metrics and Cohen’s
kappa score, as well as qualitative semantic consistency. At last, in Section
III-C we introduce a novel superpixel merge method that uses the Malahanobis
[21] distance to merge neighboring superpixels enforcing a minimum pixel count
for each segment.

###  III-A OpenGMM

Figure 2: An example showing how different objects can be represented by
different distributions. Due to the multimodal representation capability, GMM
is better suited for representing real-world data than Principal Component
Scoring [13].

OpenGMM builds upon the previously proposed OpenPCS [13] method, OpenPCS uses
Principal Component Analysis (PCA) to compress the representation extracted
from the backbones and uses the generated representation to do the OOD pixel
detection. OpenGMM replaces PCA with Gaussian Mixture Model (GMM) [15]. GMM’s
multimodal representation should be better suited for modeling real-world
pixels that may not conform to unimodal representations.

Like OpenPCS, OpenGMM takes into account intermediate feature maps together
with the last layer activation map. Combining the activations from the earlier
layers with the last layers produces a tensor that fuses low and high semantic
level information. The concatenated tensor may have hundreds or thousands of
channels, which is known to be redundant [53, 54]. OpenGMM handles the
concatenated tensor size and redundancy of activations by fitting a Gaussian
Mixture of Models (GMM) rather than fitting a Principal Component Analysis
(PCA) like OpenPCS. The GMM is applied to model each closed set KKC
distribution, generating as many models as KKCs. Each GMM model generates a
score tensor with the log-likelihood values for all pixels, a final score
tensor is generated combining all GMM scores with the closed set prediction.
To determine the OOD pixels, the final score is evaluated and pixels below the
threshold are set to unknown.

The OpenPCS framework proposed by Oliveira et al. [13] and, as an extention,
OpenGMM allow for multiple closed set backbones. We adopt three backbones for
OpenGMM and OpenPCS: DenseNet-121 (DN-121) [54], WideResNet-50 (WRN-50) [55]
and U-net [7].

Readers should notice that adapting any closed set semantic segmentation
network to the OpenGMM and OpenPCS frameworks is relatively quick and simple
to implement, without requiring retraining or additional layers to be trained.
This is in contrast to other existing OSR/OSS methods [56, 57, 53, 22].

The plug-and-play characteristic of methods such as OpenPCS and OpenGMM is a
great advantage when considering the problem of adapting the solution to real-
world applications and novel domains.

###  III-B Improving Semantic Consistency with Superpixels

Superpixels are usually employed before or during the segmentation process
[23, 24, 25, 26, 29]. In general, when employed as post processing, normally
they use the input image to generate the SPS and apply it somehow in the
output prediction to generate more consistent borders between objects and
improve semantic consistency. Following the literature, in the present work we
employ superpixels as a post-processing step applied to the OSS scores.

As an output, OSS methods generate a tensor of the same size as the input
image containing the scores or reconstruction errors depending on the method
used. This final tensor is then processed with the superpixels and all pixels
are set to the mean value of the segment from which they belong. Algorithm 1
details the usage of the superpixel oversegmentation in the final step just
before the open set recognition phase. While our post-processing scheme is
agnostic to the choice of SPS algorithm, for this study we evaluated SLIC
[41], Quickshift (QS) [38] and Felzenszwalb (FZ) [18]. These algorithms were
chosen because they have different generation characteristics and different
pros and cons, as detailed in Section II.

Algorithm 1 The output of the OSS method is post-processed using the
superpixels segmentation. All pixels of a given segment assume the mean value
of all pixels in that segment.

1:scores $\triangleright$ pixelwise array

2:segments $\triangleright$ list of segments

3:procedure post_process($scores,segments$)

4: pred = zeros(scores.size)

5: for $segment\in Segments$ do

6: pred[segment] = mean(scores[segment])

7: end for

8: return pred

9:end procedure

The remaining steps of the open set recognition process are kept the same for
each OSS method. The superpixel oversegmentations are homogeneous and tend to
respect object borders. Applying the superpixels to the score image smooths
the segmented areas, aiding the OSS algorithm in avoiding errors within the
segmented objects.

Each superpixel algorithm has its own generation characteristics, and the
final segmentation reproduces these particularities in the results. We can see
in Figure 4 an illustrative example of two SPS that present different
characteristics and may represent better different scenarios. In this example,
the SLIC algorithm could better represent textures, while the FZ algorithm
could better identify borders, although none of the single SPS could represent
the underlying image properly, a great improvement is observed when FuSS is
used.

###  III-C FuSS

All SPS algorithms share the same main goals, namely to generate homogeneous
areas and to respect borders between different objects. In other words, their
goal is to minimize the intracluster/intrasegment variance while maximizing
the intercluster/intersegment variance.

Figure 3: A toy example illustrating the workflow to merge two different
superpixel segmentations. First the input image $x$ is processed by 2
different superpixel segmentation algorithms (Alg. 1 and Alg. 2). Then the
generated segmentations $s_{{}_{1}}$ and $s_{{}_{2}}$ are merged into the
final segmentation $s_{{}_{FuSS}}$ using the merging procedure described in
Algorithm 2. Figure 4: Comparison of the resulting segmentation from two SPS
algorithms (Felzenszwalb and SLIC) and our proposed fusion algorithm, FuSS.
The first row shows the input image superimposed with the superpixel segments
and the second row depicts the closer class fit of each segment according to
the real labels. Red arrows indicate areas where class boundaries failed when
using one single SPS algorithm, while gray arrows point to these same regions
fixed after applying the FuSS algorithm.

Different SPS algorithms use distinct operations to produce the final
segmentation, and as a result experience distinct failure cases. The idea
behind FuSS is to fuse segments from multiple types of superpixel generation
algorithms. The motivation for the use of distinct families of SPS methods is
to take advantage of the different generation characteristics, enhancing the
advantages and mitigating the disadvantages of each method. Figure 4 shows an
example of how each single SPS represents ground truth and compare to FuSS
segmentation. To generate a ground truth that perfectly overlaps with all
segments we assign the class with highest pixel count to the entire segment
for each segment. We can observe through the qualitative result that the FuSS
improves the representation of the image in relation to the ground truth.

The first step of the fusion procedure is to generate unique segments
superposing two different segmentations. This procedure is prone to produce
some extremely small segments. Thus, to deal with this unwanted side-effect,
we use the Malahanobis distance [21] to fuse the closest neighbor segments
until there are no more segments below the specified pixel minimum size. FuSS
is detailed in Algorithm 2.

Figure 3 illustrates two different superpixels methods being merged. As shown
in the figure, the final SPS respects both segmentations’ borders, and each
segment depict better the underlying region. FuSS is agnostic to the SPS
algorithm, being applicable to any set of distinct superpixel algorithms.
However, in practice, using more than two algorithms yields exceedingly small
segments, motivating our experiments to focus only on pairs of algorithms.

Algorithm 2 Pseudo-algorithm detailing the auxiliary procedure of joining
segmentations used in FuSS.

1:scores $\triangleright$ pixelwise array

2:segments $\triangleright$ list of segments

3:procedure join_segmentations($seg1,seg2$)

4: joint = []

5: for $s1\in seg1$ do

6: $\triangleright$ Selecting s2 $\in$ seg2 where s2 $\cap$ s1 $\neq\emptyset$

7: for s2 $\in$ seg2.overlap_segments(s1) do

8: overlap_area = $s1\cap s2$

9: joint.add_new_segment(overlap_area)

10: end for

11: end for

12: return joint

13:end procedure

14:

15:procedure FuSS($seg1,seg2$)

16: joint = join_segmentations(seg1, seg2)

17: for $s\in joint$ do

18: if $s.size<min\\_size$ then

19: closest = closest_neighbor(s, joint)

20: joint = merge_segments(joint, s, closest)

21: end if

22: end for

23: return joint

24:end procedure

##  IV Experimental Setup

We used PyTorch [58] to implement all neural network models and backbones,
with a NVIDIA Titan X with 12GB of memory. All models fit on a single graphics
card, filling between 10 GB and 11 GB of memory. SPS algorithms were
implemented using the scikit-learn444https://scikit-learn.org/ library. The
official implementation for FuSS is publicly available for encouraging
reproducibility.

We tested OpenGMM with 4, 8 and 16 components, resulting in minimal
performance differences across this range of values. Hence, all OpenGMM
results reported in this section were run with 4 components. For OpenPCS we
adopted the 16 components from the original implementation.

###  IV-A Datasets and Evaluation Protocol

We employed the Leave One Class Out (LOCO) protocol used by Oliveira et al.
[13] in this work to emulate an open set scenario, since the two selected
datasets were intended to closed set semantic segmentation. The LOCO protocol
splits the known classes and selects one of them to be ignored during
training, allowing the evaluation of open set methods on the hidden class.
Thus, we only backpropagate the loss of pixels from known classes, ignoring
the background, borders, miscellaneous and unknown classes.

We selected the International Society for Photogrammetry and Remote Sensing
(ISPRS) 2D Semantic
Labeling555https://www.isprs.org/education/benchmarks/UrbanSemLab/default.aspx
datasets of Vaihingen and Potsdam for our experiments. Both datasets were
previously used in OSS scenarios [32, 13, 22]. Vaihingen images present a
9cm/pixel spatial resolution, varying from 2000 to 2500 pixels per axis, while
Potsdam samples have a 5cm/pixel spatial resolution and $6000\times 6000$
pixels each. Both datasets contain 6 KKCs: impervious surfaces, buildings, low
vegetation, high vegetation, car and miscellaneous; and 1 known unknown class
(KUC): segmentation boundaries between objects. Among KKCs, the miscellaneous
class is composed mostly of areas presenting image acquisition noise and
objects unimportant to practical remote sensing applications, motivating its
removal from our experimental procedure. We used the same bands employed in
previous works on OSS [13, 22], namely IR-R-G-nDSM.

Both datasets were separated into 3 sets each: training, validating and
testing. For Vaihingen the selected patches were: 1, 3, 5, 7, 13, 17, 21, 26,
32 and 37 for training; 11, 15, 28, 30 and 34 for testing; and 23 for
validation. For Potsdam the selected patches were: 2_10, 2_13, 2_14, 3_10,
3_12, 3_13, 3_14, 4_11, 4_12, 4_13, 4_14, 4_15, 5_10, 5_12, 5_13, 5_14, 5_15,
6_8, 6_9, 6_10, 6_11, 6_12, 6_13, 6_15, 7_7, 7_9, 7_11, 7_12 and 7_13 for
training; 2_11, 2_12, 4_10, 5_11, 6_7, 7_8 and 7_10 for testing; and 3_11 and
6_14 for validation.

###  IV-B Backbones

All OSS models use a closed set backbone as a starting point to identify the
OOD pixels. Relying on previous results [32, 13], we used three of the best
performing backbones for our experiments: U-Net [7], Wide ResNet-50 (WRN50)
[55] and DenseNet-121 (DN121) [54]. For CoReSeg [22], only U-Net was used,
since it naturally matches the structural constraint of this method.

###  IV-C Metrics

We use the Receiver Operating Characteristic (ROC) curve and the Area Under
the ROC (AUROC) improvements as evidence that including superpixels in the
inference phase improves OOD recognition. To measure the performace for both
KKCs and UUCs together we also employed the Cohen’s kappa score [59]
($\kappa$). The use of $\kappa$ allows the assessment of the reliability of
the process since it measures the agreement of the methods with the ground
truth. Both metrics were used to compare the predictions between the methods
with and without superpixel post-processing.

###  IV-D Superpixel Configurations

We conducted experiments with 70 different superpixel configurations for the
Vaihingen ablation study. From the initial ablation, we selected the 6 best
configurations on Vaihingen to run in Potsdam, which is a much larger dataset.
The 70 configurations were divided into 2 categories:

  1. 1.

single – an execution with different parameters of the selected SPS algorithm;

  2. 2.

FuSS – merges 2 different single configurations using the proposed procedure.

For the subset of the 70 superpixel settings that used FuSS, we tested the use
of mean and median when calculating the distance between neighboring
superpixel regions. Furthermore, we tested 2 different minimum sizes for the
superpixels: 25 and 50 pixels. The implemented SPS algorithms often use the
mean to represent each segment or to define the SPS. Thus, we tested the
median to find out if it could better represent the image bands.

In the Appendix A we list the 11 superpixel generation configurations reported
in this work both in Sections V and VI. The same merge parameters were used
for the FuSS configurations presented in Section VI: mean as distance to merge
superpixels and with a minimum size of 50 pixels.

| UUC: imp. surf. | UUC: building | UUC: low veg. | UUC: high veg. | UUC: car | Average
---|---|---|---|---|---|---
Type | min | avg. | max | min | avg. | max | min | avg. | max | min | avg. | max | min | avg. | max | min | avg. | max
single | .860 | .898 | .912 | .934 | .956 | .961 | .702 | .730 | .740 | .824 | .873 | .886 | .709 | .872 | .918 | .830 | .866 | .878
FuSS | .903 | .907 | .911 | .957 | .960 | .961 | .729 | .735 | .740 | .878 | .884 | .887 | .813 | .884 | .913 | .860 | .874 | .888
TABLE I: The table presents the results aggregated by type of superpixel
generation. Two types of generation were used, “single” and “FuSS”. “Single”
stands for a generation using one superpixel method alone, while “FuSS” stands
for the procedure presented in Section III-C. The columns min, avg. and max
stand for the minimum, average and maximum AUROC value among all superpixel
configurations for each type of generation. The last block is the is the
average value among all tested UUCs.

##  V Ablation Study

UUCs | Average | Superpixel All | Value Used
---|---|---|---
Imp. Surf. | Building | Low Veg. | High Veg. | Car
.903 | .958 | .737 | .881 | .897 | $\mathbf{.8752\pm.083}$ | Yes | Mean
.906 | .959 | .730 | .880 | .865 | $.8676\pm.085$ | Yes | Median
.903 | .958 | .737 | .881 | .897 | $\mathbf{.8752\pm.083}$ | No | Mean
.906 | .959 | .730 | .880 | .865 | $.8677\pm.085$ | No | Median
TABLE II: The table shows the AUROC results aggregated by the use in post-
processing. The columns ”superpixel All“ indicates if the SPS was applied to
the entire predictions or only to the scores and the ”Value Used” column shows
which value is used to represent the entire superpixel. Figure 5: Comparison
of minimum, average and maximum AUROC for all superpixel configurations
generated for the ablation study on CoReSeg [22] with the Vaihingen dataset.
The barplot shows single superpixel algorithms (yellow) and FuSS (blue) for
the average across all UUCs in the LOCO protocol. Confidence Intervals (CIs)
according to a paired two-tailed t-Student test with $p\leq 0.05$ across the 5
classes are shown as error bars, highlighting the statistical significance of
employing FuSS instead of single SPS algorithms. The lower y-axis is trimmed
to 0.5 for better visualization of the CIs.

Besides the 70 different configurations mentioned in Section IV, we applied
the superpixels in 4 distinct manners, adding up to 280 different
configurations. We executed the ablation study for the Vaihingen dataset using
CoReSeg [22] which has a fixed architecture. OpenPCS and OpenGMM can easily
switch backbones and already have 3 backbones tested each, to run the ablation
tests for OpenGMM and OpenPCS would be prohibitive as 280 tests would need to
be run for each configuration.

We split the following section into two sets of experiments: 1) SPS generation
(Section V-A), and 2) Post-Processing for OSS (Section V-B).

###  V-A Superpixel Generation

Table I and Figure 6 present the results aggregated by generation strategy.
The results shows that FuSS achieved considerably stabler results in average
and regardless of the UUC as well. A key point to use superpixels is to select
good parameters for the superpixel generation. FuSS combines different SPSs
and made the selection of parameters less relevant to the process. Thus,
despite the parameter selection, the combination of 2 different
oversegmentations generates a more reliable result. Figure 5 presents the
results for the minimum, average and maximum results for both strategies of
generation. The error bars show the confidence intervals according to a paired
two-tailed t-Student test with $p\leq 0.05$ across all 5 tested scenarios.

The use of mean or median to represent the entire superpixel when calculating
the distances had no influence in the results, as the superpixel resultant
segmentations were similar or even equal in all cases. The same behavior was
perceived when considering the minimum size for the segments. This behavior
implies that the objects may be represented with either 25 or 50 pixels as the
value of minimum size. We can see at Table VIII in Appendix A a list with the
best 20 results among all executed tests for this ablation study.

Regarding the FuSS procedure, the ablation study performed in Vaihingen shows
that the method achieve stabler results. The worst and the best results are
closer when compared to using a single superpixel method as shown in Figure 6.
When using any superpixel method, a key part of the process is the selection
of the parameters. In this regard, using FuSS relieves this burden, since it
combines two different methods and generates segments that fits the input
image better. Based on our observations, a conjecture to be studied further is
that the more different the input SPSs are, the better the final merged
superpixel segmentation is.

###  V-B Post-Processing for OSS

Regarding the use of the superpixel segmentation to improve the results of the
OSS methods, we tested 4 different ways:

  * •

applying to entire closed set segmentation and to the scores/reconstruction
errors;

  * •

applying only to the scores/reconstruction errors;

  * •

use the mean to represent the score/reconstruction error for the entire
superpixel;

  * •

use the median to represent the score/reconstruction error for the entire
superpixel.

Table II presents the results aggregated by the ways superpixels were used in
the OSS post-process. The best results were achieve using the mean value of
the score/reconstruction error to represent the superpixel. Applying the
superpixel only to the scores/reconstruction errors or along the closed set
prediction resulted in the same performance, implying that the variations are
equivalent.

Figure 6: Boxplot with the AUROC results for all configurations used for the
ablation study for CoReSeg in the Vaihingen dataset. The boxplot shows both
single superpixel algorithms (blue) and FuSS (yellow) for individual UUCs and
for the average.

##  VI Results and Discussion

|  | Vaihingen | Potsdam
---|---|---|---
|  | UUCs |  | UUCs |
Backbone | OSS method | 0 | 1 | 2 | 3 | 4 | Average | 0 | 1 | 2 | 3 | 4 | Average
DN121 | OpenGMM | .872 | .936 | .646 | .688 | .687 | $\mathbf{.7658\pm.129}$ | .829 | .907 | .642 | .553 | .866 | $\mathbf{.7594\pm.154}$
DN121 | OpenPCA | .860 | .925 | .643 | .708 | .650 | $.7572\pm.128$ | .841 | .901 | .546 | .569 | .882 | $.7478\pm.175$
U-Net | CoReSeg | .884 | .934 | .710 | .867 | .854 | $\mathbf{.8499\pm.084}$ | .824 | .901 | .698 | .631 | .768 | $\mathbf{.7644\pm.106}$
U-Net | OpenGMM | .908 | .787 | .522 | .846 | .601 | $.7328\pm.165$ | .870 | .856 | .371 | .569 | .858 | $.7048\pm.226$
U-Net | OpenPCA | .899 | .799 | .508 | .843 | .538 | $.7174\pm.181$ | .875 | .859 | .423 | .531 | .835 | $.7046\pm.212$
WRN50 | OpenGMM | .885 | .911 | .611 | .648 | .619 | $\mathbf{.7348\pm.150}$ | .851 | .870 | .403 | .453 | .824 | $.6802\pm.231$
WRN50 | OpenPCA | .854 | .873 | .628 | .658 | .622 | $.7270\pm.126$ | .853 | .880 | .424 | .462 | .840 | $\mathbf{.6918\pm.228}$
TABLE III: For each combination of backbone and OSS method the table shows the
classwise AUROC and average (Avg.) AUROC in Vaihingen and Potsdam. Numbered
columns stand for: 1 – Impervious Surfaces, 2 – Building, 3 – Low Vegetation,
4 – High Vegetation and 5 – Car

###  VI-A OpenGMM Comparisons

Table III presents comparisons between OpenGMM, OpenPCS and CoReSeg with the
same closed set backbones. In Table III for each backbone the cells in bold
shows the baseline results before any post-processing. The post-processing
results will be discussed in Section VI-B.

For the Vaihingen dataset we can observe that in all tested cases OpenGMM
outperformed OpenPCS with the same backbone, while Potsdam yielded mixed
results, with OpenGMM and OpenPCS presenting overall similar performances and
even surpassing OpenPCS in 4 out of 6 direct comparisons. We attribute the
improvement shown by OpenGMM over OpenPCS to its multimodal representation
capability for modeling real-world data, regardless of the number of KKCs. The
worse results in Potsdam are attributed mainly to OpenGMM’s poorer
performances on two UUCs: Low Vegetation and High Vegetation. The instability
of OSS algorithms in these two particular classes is known from previous
works, possibly due to the large semantic intra-class variabilities.

It is worth mentioning that our experimental procedure is not exactly the same
used by Oliveira et al. [13], therefore, all experiments were re-executed to
ensure comparability and the results presented here are different from the
previous work.

###  VI-B Post-Processing OSS with Superpixels

Superpixel post-processing improved AUROC in all tested scenarios. The
improvement and the better semantic consistency can be clearly observed in
Figures 7 and 8.

Figure 7: Qualitative results for an image from the Vaihingen dataset under
different settings of UUCs and OSS methods. The use of the proposed superpixel
post-processing method generates clearer segmentations, avoiding the usual
mislabeling of unkown pixels.  Figure 8: OSS predictions obtained using
CoReSeg for an image of the Vaihingen dataset with all tested UUCs. The last
column shows qualitatively the improvement achieved using the proposed
superpixel post-processing method. We can also observe the impact of the
method in the Reconstruction Loss.

Tables V and IV show, in terms of AUROC, the improvement observed in the
quantitative analysis. We only observe a worsening in the AUROC average values
for the cases in which the original closed set result of the OSS method was
already poor for a certain UUC. Table IV shows this for both OpenGMM and
OpenPCS for UUCs Low Vegetation and Car with U-Net as backbone.

From the results, we can state that if the OSS results presents little
semantic consistency, the post-processing is not able to consistently improve
the results. In fact, the experiments show that the post-processing can worsen
the results in such cases where prediction performance is already rather poor.
However, when the OSS model with no superpixel post-processing yields
consistent segmentations with inconsistent predictions mainly in object
borders and/or salt-and-pepper artifacts, superpixel post-processing was quite
effective. In other words, assuming that the superpixels are representative,
homogeneous and respect the edges of the image, a better base OSS result
allows the superpixel post-processing to correct mistakes inside the
superpixels, as these mistakes are usually the minority of pixels. For
instance, considering CoReSeg with U-Net as backbone and Car as the UUC, the
AUROC improved from 0.854 to 0.913 for Vaihingen, and from 0.768 to 0.816 for
Potsdam. Another clear improvement can be seen when analysing the average
AUROC for the 5 UUCs, which improved from 0.850 to 0.880 in Vaihingen and from
0.764 to 0.797 in Potsdam.

Tables VI and VII show the comparison of the Cohen’s Kappa scores for
different thresholds on the tested OSS methods. $\kappa$ improved for all
thresholds in the results presented in Table VII. Metrics followed by
$\dagger$ show statistically significant improvements with the use of
superpixels, which was achieved for 8 out of 11 thresholds in CoReSeg.
Considering the OpenPCS and OpenGMM performances presented in Table VI, the
results of thresholds lower than 0.5 were improved, but they were not
statistically significant. As for thresholds between 0.5 and 0.9, the $\kappa$
improvement was statistically significant. The proposed post-processing was
able not only to improve the AUROC threshold independent score and the
qualitative results, but the results for Cohen’s kappa score reassures that
the post-processing improves the reliability of the predictions.

|  |  | UUCs |  |
---|---|---|---|---|---
Backbone | OSS Method | Config. | Imp. Surf. | Buildings | Low Veg. | High Veg. | Car | avg. AUROC | avg. px/seg
DN121 | OpenGMM | - | .872 | .936 | .646 | .688 | .687 | $.7658\pm.129$ | -
DN121 | OpenGMM | fuss01 | .891$\dagger$ | .946 | .652$\dagger$ | .698 | .689 | $.7752\pm.133$ | 750
DN121 | OpenGMM | single02 | .880 | .954$\dagger$ | .650 | .696 | .701$\dagger$ | $\mathbf{.7762\pm.133\dagger}$ | 491
DN121 | OpenPCS | - | .860 | .925 | .643 | .708 | .650 | $.7572\pm.128$ | -
DN121 | OpenPCS | fuss02 | .878 | .948 | .651 | .720 | .636 | $.7694\pm.135$ | 322
DN121 | OpenPCS | single02 | .876 | .951 | .649 | .718 | .663 | $.7714\pm.135$ | 491
U-Net | CoReSeg | - | .884 | .934 | .710 | .867 | .854 | $.8499\pm.084$ | -
U-Net | CoReSeg | single03 | .906 | .958 | .737 | .884 | .903 | $.8776\pm.083$ | 476
U-Net | CoReSeg | fuss03 | .906 | .959 | .739 | .886 | .910 | $\mathbf{.8800\pm.083}$ | 434
U-Net | OpenGMM | - | .908 | .787 | .522 | .846 | .601 $\dagger$ | $.7328\pm.165$ | -
U-Net | OpenGMM | fuss01 | .930$\dagger$ | .805 | .512 | .846 | .597 | $.7380\pm.176$ | 750
U-Net | OpenGMM | single01 | .929 | .802 | .519$\dagger$ | .848 | .599 | $.7394\pm.173\dagger$ | 322
U-Net | OpenPCS | - | .899 | .799 | .508 | .843 | .538 | $.7174\pm.181$ | -
U-Net | OpenPCS | single04 | .925 | .814 | .501 | .848 | .527 | $.7230\pm.195$ | 322
U-Net | OpenPCS | fuss02 | .925 | .813 | .502 | .852 | .526 | $.7236\pm.196$ | 492
WRN50 | OpenGMM | - | .885 | .911 | .611 | .648 | .619 | $.7348\pm.150$ | -
WRN50 | OpenGMM | single01 | .904 | .936$\dagger$ | .622 | .665 | .618 | $.7490\pm.165$ | 491
WRN50 | OpenGMM | fuss04 | .918$\dagger$ | .936$\dagger$ | .630 | .667 |  .641 $\dagger$ |  $\mathbf{.7584\pm.155}$ $\dagger$ | 698
WRN50 | OpenPCS | - | .854 | .873 | .628 | .658 | .622 | $.7270\pm.126$ | -
WRN50 | OpenPCS | fuss04 | .896 | .915 | .628 | .681 | .571 | $.7378\pm.158$ | 698
WRN50 | OpenPCS | single02 | .890 | .912 | .648 | .681 | .600 | $.7462\pm.144$ | 491
TABLE IV: Results for the Vaihingen dataset ordered by the average AUROC for each pair Backbone/OSS Method. Each configuration of the column Superpixel Config. is better detailed in the Appendix A. The column avg. AUROC shows the average AUROC between the UUCs; the column avg. px/seg shows the average size of the segments generated by the Superpixel Config. The $\dagger$ symbol marks when results for OpenGMM is better than OpenPCS with the same Backbone-OSS method. |  |  | UUCs |  |
---|---|---|---|---|---
Backbone | OSS Method | Config. | Imp. Surf. | Buildings | Low Veg. | High Veg. | Car | avg. AUROC | avg. px/seg
DN121 | OpenGMM | - | .829 | .907 | .642 | .553 | .866 | $.7594\pm.154$ | -
DN121 | OpenGMM | single03 | .852 | .914 | .642$\dagger$ | .559 | .871 | $.7676\pm.154$ | 509
DN121 | OpenGMM | fuss01 | .848 | .916$\dagger$ | .642$\dagger$ | .567 | .887 | $\mathbf{.7720\pm.157}\dagger$ | 1006
DN121 | OpenPCS | - | .841 | .901 | .546 | .569 | .882 | $.7478\pm.175$ | -
DN121 | OpenPCS | single01 | .866 | .913 | .569 | .571 | .888 | $.7614\pm.177$ | 876
DN121 | OpenPCS | fuss01 | .863 | .913 | .558 | .582 | .904 | $.7640\pm.178$ | 1006
U-Net | CoReSeg | - | .824 | .901 | .698 | .631 | .768 | $.7644\pm.106$ | -
U-Net | CoReSeg | single05 | .860 | .913 | .738 | .654 | .813 | $.7956\pm.102$ | 1447
U-Net | CoReSeg | fuss01 | .866 | .911 | .733 | .659 | .817 | $\mathbf{.7972\pm.102}$ | 1006
U-Net | OpenGMM | - | .870 | .856 | .371 | .569 | .858 | $.7048\pm.226$ | -
U-Net | OpenGMM | single03 | .898 | .880 | .357 | .571 | .876 | $.7164\pm.243$ | 509
U-Net | OpenGMM | fuss01 | .895 | .878 | .361 | .578$\dagger$ | .897$\dagger$ | $.7218\pm.243$ | 1006
U-Net | OpenPCS | - | .875 | .859 | .423 | .531 | .835 | $.7046\pm.212$ | -
U-Net | OpenPCS | single03 | .901 | .893 | .415 | .524 | .864 | $.7194\pm.232$ | 509
U-Net | OpenPCS | fuss04 | .902 | .892 | .416 | .526 | .874 | $.7220\pm.233$ | 876
WRN50 | OpenGMM | - | .851 | .870 | .403 | .453 | .824 | $.6802\pm.231$ | -
WRN50 | OpenGMM | single03 | .884 | .890 | .418 | .455 | .835 | $.6964\pm.239$ | 509
WRN50 | OpenGMM | fuss01 | .880 | .890 | .413 | .457 | .853 | $.6986\pm.242$ | 1006
WRN50 | OpenPCS | - | .853 | .880 | .424 | .462 | .840 | $.6918\pm.228$ | -
WRN50 | OpenPCS | single03 | .888 | .904 | .423 | .457 | .856 | $.7058\pm.243$ | 509
WRN50 | OpenPCS | fuss01 | .884 | .904 | .419 | .457 | .873 | $\mathbf{.7074\pm.247}$ | 1006
TABLE V: Results for the Potsdam dataset ordered by the average AUROC for each pair Backbone/OSS Method. Each configuration of the column Superpixel Config. is better detailed in the Appendix A. The column avg. AUROC shows the average AUROC between the UUCs; the column avg. px/seg shows the average size of the segments generated by the Superpixel Config. The $\dagger$ symbol marks when results for OpenGMM is better than OpenPCS with the same Backbone. Method | UUC | FuSS | Thresholds
---|---|---|---
0.00 | 0.10 | 0.20 | 0.30 | 0.40 | 0.50 | 0.60 | 0.70 | 0.80 | 0.90
OpenGMM | Imp. Surf. | No | 0.500 | 0.500 | 0.510 | 0.568 | 0.616 | 0.644 | 0.665 | 0.670 | 0.642 | 0.541
OpenGMM | Imp. Surf. | Yes | 0.500 | 0.530 | 0.560 | 0.591 | 0.621 | 0.649 | 0.676 | 0.693 | 0.687 | 0.617
OpenGMM | Buildings | No | 0.461 | 0.461 | 0.481 | 0.532 | 0.581 | 0.608 | 0.631 | 0.649 | 0.654 | 0.612
OpenGMM | Buildings | Yes | 0.461 | 0.493 | 0.524 | 0.555 | 0.586 | 0.615 | 0.643 | 0.664 | 0.678 | 0.652
OpenGMM | Low Veg. | No | 0.606 | 0.575 | 0.567 | 0.551 | 0.525 | 0.490 | 0.436 | 0.365 | 0.278 | 0.177
OpenGMM | Low Veg. | Yes | 0.606 | 0.564 | 0.558 | 0.541 | 0.525 | 0.496 | 0.462 | 0.405 | 0.325 | 0.218
OpenGMM | High Veg. | No | 0.518 | 0.509 | 0.523 | 0.534 | 0.536 | 0.520 | 0.484 | 0.418 | 0.304 | 0.166
OpenGMM | High Veg. | Yes | 0.518 | 0.512 | 0.526 | 0.537 | 0.539 | 0.525 | 0.498 | 0.443 | 0.368 | 0.217
OpenGMM | Car | No | 0.775 | 0.747 | 0.712 | 0.672 | 0.607 | 0.515 | 0.393 | 0.281 | 0.175 | 0.073
OpenGMM | Car | Yes | 0.775 | 0.735 | 0.705 | 0.655 | 0.573 | 0.500 | 0.414 | 0.324 | 0.223 | 0.131
OpenGMM | Average | No | 0.572 | 0.558 | 0.559 | 0.571 | 0.573 | 0.555 | 0.522 | 0.477 | 0.411 | 0.314
OpenGMM | Average | Yes | 0.572 | 0.567 | 0.575 | 0.576 | 0.569 | 0.557 | 0.539$\dagger$ | 0.506$\dagger$ | 0.456$\dagger$ | 0.367$\dagger$
OpenPCS | Imp. Surf. | No | 0.500 | 0.525 | 0.552 | 0.578 | 0.602 | 0.622 | 0.635 | 0.634 | 0.607 | 0.519
OpenPCS | Imp. Surf. | Yes | 0.500 | 0.522 | 0.550 | 0.583 | 0.613 | 0.638 | 0.656 | 0.661 | 0.648 | 0.575
OpenPCS | Buildings | No | 0.461 | 0.466 | 0.520 | 0.543 | 0.566 | 0.586 | 0.599 | 0.605 | 0.598 | 0.551
OpenPCS | Buildings | Yes | 0.461 | 0.472 | 0.524 | 0.551 | 0.579 | 0.607 | 0.627 | 0.632 | 0.640 | 0.603
OpenPCS | Low Veg. | No | 0.606 | 0.606 | 0.604 | 0.589 | 0.547 | 0.478 | 0.406 | 0.332 | 0.293 | 0.245
OpenPCS | Low Veg. | Yes | 0.606 | 0.590 | 0.592 | 0.585 | 0.562 | 0.532 | 0.463 | 0.367 | 0.310 | 0.254
OpenPCS | High Veg. | No | 0.518 | 0.511 | 0.529 | 0.541 | 0.542 | 0.527 | 0.489 | 0.424 | 0.319 | 0.168
OpenPCS | High Veg. | Yes | 0.518 | 0.517 | 0.541 | 0.554 | 0.556 | 0.540 | 0.505 | 0.456 | 0.379 | 0.213
OpenPCS | Car | No | 0.775 | 0.771 | 0.763 | 0.732 | 0.623 | 0.468 | 0.369 | 0.293 | 0.198 | 0.109
OpenPCS | Car | Yes | 0.775 | 0.748 | 0.723 | 0.674 | 0.573 | 0.443 | 0.332 | 0.237 | 0.154 | 0.086
OpenPCS | Average | No | 0.572 | 0.576 | 0.594 | 0.597 | 0.576 | 0.536 | 0.500 | 0.458 | 0.403 | 0.319
OpenPCS | Average | Yes | 0.572 | 0.570 | 0.586 | 0.590 | 0.576 | 0.552$\dagger$ | 0.517$\dagger$ | 0.471$\dagger$ | 0.426$\dagger$ | 0.346$\dagger$
TABLE VI: Kappa scores with thresholds varying from 0.0 to 1.0 for the Vaihingen dataset with OpenGMM and OpenPCS as the OSS method and WRN50 as backbone. The third column represents wether the superpixel post-processing was applied. The last two rows for each combination of backbone and OSS method shows the average kappa value across all UUCs. For the average rows, the $\dagger$ symbol means that the results has statistical significance, according to a paired two-tailed t-Student test with $p\leq 0.05$ across the 5 classes. UUC | FuSS | Thresholds
---|---|---
0.90 | 0.91 | 0.92 | 0.93 | 0.94 | 0.95 | 0.96 | 0.97 | 0.98 | 0.99 | 1.00
Imp. Surfaces | No | 0.678 | 0.683 | 0.687 | 0.692 | 0.696 | 0.699 | 0.701 | 0.698 | 0.685 | 0.564 | 0.540
Imp. Surfaces | Yes | 0.703 | 0.705 | 0.708 | 0.710 | 0.710 | 0.710 | 0.712 | 0.711 | 0.700 | 0.549 | 0.540
Buildings | No | 0.692 | 0.698 | 0.703 | 0.707 | 0.710 | 0.709 | 0.705 | 0.695 | 0.674 | 0.623 | 0.504
Buildings | Yes | 0.744 | 0.746 | 0.748 | 0.749 | 0.747 | 0.742 | 0.733 | 0.717 | 0.689 | 0.626 | 0.504
Low Vegetation | No | 0.587 | 0.589 | 0.591 | 0.592 | 0.594 | 0.595 | 0.597 | 0.600 | 0.605 | 0.611 | 0.621
Low Vegetation | Yes | 0.600 | 0.601 | 0.604 | 0.603 | 0.605 | 0.606 | 0.607 | 0.609 | 0.611 | 0.615 | 0.621
High Vegetation | No | 0.656 | 0.657 | 0.658 | 0.658 | 0.656 | 0.653 | 0.648 | 0.638 | 0.619 | 0.555 | 0.555
High Vegetation | Yes | 0.683 | 0.684 | 0.683 | 0.683 | 0.681 | 0.676 | 0.665 | 0.645 | 0.609 | 0.549 | 0.555
Car | No | 0.677 | 0.684 | 0.692 | 0.701 | 0.710 | 0.720 | 0.732 | 0.745 | 0.757 | 0.769 | 0.778
Car | Yes | 0.713 | 0.719 | 0.726 | 0.733 | 0.741 | 0.749 | 0.758 | 0.764 | 0.769 | 0.773 | 0.779
Average | No | 0.658 | 0.662 | 0.666 | 0.670 | 0.673 | 0.675 | 0.677 | 0.675 | 0.644 | 0.649 | 0.600
Average | Yes |  0.689 $\dagger$ | 0.691$\dagger$ | 0.694$\dagger$ | 0.695$\dagger$ | 0.697$\dagger$ | 0.697$\dagger$ | 0.695$\dagger$ | 0.689$\dagger$ | 0.645 | 0.652 | 0.600
TABLE VII: Kappa scores with thresholds varying from 0.9 to 1.0 for the
Vaihingen dataset with CoReSeg as the OSS method. The second column represents
wether the superpixel post-processing was applied. For the average rows, the
$\dagger$ symbol means that the results has statistical significance,
according to a paired two-tailed t-Student test with $p\leq 0.05$ across the 5
classes.

##  VII Conclusion

In this paper, we presented two different approaches to improve the semantic
consistency for the results of OSS. The first one is an extension of OpenPCS
[13] by replacing the unimodal Principal Component model by a multimodal
Gaussian Mixture of Models. The second one is a superpixel post-processing
pipeline capable of benefiting OSS predictions based on single SPS algorithms
or a mixture of them in a novel SPS fusion method named FuSS.

OpenGMM improved OpenPCS results for most of the UUCs and most of tested
backbones in terms of AUROC average value. Furthermore, the superpixel post-
processing method achieved state-of-the-art results for both Vaihingen and
Potsdam datasets when applied to CoReSeg, improving the average AUROC in 0.030
for Vaihingen and 0.033 for Potsdam. The improvement produced by the post-
processing was statistically significant in the majority of tested cases.

Our novel FuSS fusion scheme uses Malahanobis distance to merge neighboring
segments below the established minimum size limit. FuSS produced more stable
and reliable SPS compared to the use of a single superpixel method.
Furthermore, in most cases the best results were achieved using the final
segmentation generated by FuSS. FuSSwas also proven to relieve the burden of
parameter selection for superpixel generation.

Our proposed SPS post-processing method improved the results in all tested
scenarios and for all OSS methods and backbones. This study shows that methods
aiming to improve semantic consistency can benefit from a superpixel post-
processing procedure, which helps in delimiting objects and borders.

Future works include: evaluating other contour recognition post-processing
methods; training a deep neural network that generates superpixels
simultaneously with the semantic segmentation task [60]; testing different
modern state-of-the-art segmentation backbones [61] with OpenPCS and OpenGMM;
employing attention techniques for the improvement of the OSS; and using
conditional random fields as the post processing step at the end of the
networks.

##  Appendix A Superpixel Configurations

Table VIII is complimentary to SectionV and shows the top 20 results for
Vaihingen dataset and CoReSeg [22] among the 280 tests executed. The first
line shows the baseline without post-processing, and the subsequent lines are
sorted by Average AUROC. The second column stands for the value used to
represent the superpixel for post-processing, and the third column shows the
minimum pixel count of each superpixel used by FuSS.

Below the list of all superpixels configurations used to run all tests in this
work presented in Sections VI and V and in Table VIII. FZ stands for the
Felzenszwalb et al. [18] algorithm, QS stands for the Quickshift [38]
algorithm, and SLIC stands for the method with the same name proposed by
Achanta et al. [41]:

  1. 1.

single01: FZ (scale: 100, sigma: 0.5, min_size: 50)

  2. 2.

single02: FZ (scale: 200, sigma: 0.5, min_size: 50)

  3. 3.

single03: SLIC (n_segments: n_pixels $\div$ 350, compactness: 5, $\sigma$: 1)

  4. 4.

single04: FZ (scale: 50, sigma: 0.5, min_size: 50)

  5. 5.

single05: FZ (scale: 100, sigma: 0.5, min_size: 100)

  6. 6.

fuss01:

     * •

SLIC (n_segments: n_pixels $\div$ 2000, compactness: 5, $\sigma$: 1)

     * •

FZ (scale: 200, $\sigma$: 0.7, min_size: 200)

  7. 7.

fuss02:

     * •

SLIC (n_segments: n_pixels $\div$ 1500, compactness: 5, $\sigma$: 1)

     * •

FZ (scale: 100, $\sigma$: 0.7, min_size: 150)

  8. 8.

fuss03:

     * •

SLIC (n_segments: n_pixels $\div$ 1000, compactness: 5, $\sigma$: 1)

     * •

FZ (scale: 100, $\sigma$: 0.7, min_size: 150)

  9. 9.

fuss04:

     * •

FZ (scale: 200, $\sigma$: 0.7, min_size: 200)

     * •

QS (kernel_size: 5, max_dist: 50, ratio: 0.5)

  10. 10.

fuss05:

     * •

FZ (scale: 200, $\sigma$: 0.7, min_size: 200)

     * •

QS (kernel_size: 4, max_dist: 50, ratio: 0.5)

  11. 11.

fuss06:

     * •

FZ (scale: 200, $\sigma$: 0.7, min_size: 200)

     * •

QS (kernel_size: 3, max_dist: 50, ratio: 0.5)

Config | Value Used | min. size | UUCs | avg. AUROC |
---|---|---|---|---|---
Imp. Surf. | Buildings | Low Veg. | High Veg. | Car |
- | - | - | .884 | .934 | .710 | .867 | .854 | $.8499\pm.084$ |
fuss03 | mean | 50 | 906 | .959 | .739 | .886 | .910 | $\mathbf{.8800\pm.083}$ |
fuss03 | median | 50 | .906 | .959 | .739 | .886 | .910 | $\mathbf{.8800\pm.083}$ |
fuss03 | mean | 25 | .906 | .959 | .739 | .886 | .910 | $.8797\pm.083$ |
fuss03 | median | 25 | .905 | .959 | .739 | .886 | .910 | $.8797\pm.083$ |
fuss02 | median | 50 | .906 | .959 | .740 | .885 | .908 | $.8796\pm.083$ |
fuss01 | mean | 50 | .904 | .960 | .739 | .882 | .913 | $.8796\pm.084$ |
fuss02 | mean | 50 | .906 | .959 | .740 | .885 | .908 | $.8796\pm.083$ |
fuss01 | median | 50 | .904 | .960 | .739 | .882 | .913 | $.8796\pm.084$ |
fuss01 | median | 25 | .904 | .960 | .738 | .882 | .912 | $.8794\pm.084$ |
fuss01 | mean | 25 | .904 | .960 | .738 | .882 | .912 | $.8794\pm.084$ |
fuss02 | median | 25 | .906 | .959 | .739 | .885 | .907 | $.8794\pm.083$ |
fuss02 | mean | 25 | .906 | .959 | .739 | .885 | .907 | $.8793\pm.083$ |
fuss05 | median | 50 | .905 | .960 | .739 | .881 | .910 | $.8789\pm.083$ |
fuss06 | mean | 50 | .905 | .959 | .738 | .883 | .909 | $.8788\pm.083$ |
fuss05 | mean | 50 | .905 | .960 | .739 | .881 | .909 | $.8788\pm.083$ |
fuss06 | median | 50 | .905 | .959 | .738 | .883 | .908 | $.8787\pm.083$ |
fuss04 | mean | 50 | .904 | .960 | .739 | .880 | .911 | $.8786\pm.083$ |
fuss04 | median | 50 | .904 | .960 | .739 | .880 | .911 | $.8786\pm.083$ |
fuss05 | median | 25 | .904 | .959 | .739 | .881 | .909 | $.8785\pm.083$ |
fuss05 | mean | 25 | .904 | .959 | .739 | .881 | .909 | $.8785\pm.083$ |
fuss06 | mean | 25 | .905 | .959 | .738 | .883 | .907 | $.8783\pm.083$ |
fuss04 | median | 25 | .903 | .960 | .739 | .880 | .910 | $.8783\pm.083$ |
TABLE VIII: Top 20 average AUROC results for all five UUCs tested scenarios
achieved with different superpixel post-processing methods for OSS, obtained
with CoReSeg [22] on the Vaihingen dataset. The first line shows the baseline
result without post-processing. The first column presents the superpixel
segmentation config used, the second shows which value was chosen to represent
the superpixel to calculate the distance between 2 superpixels, the third the
minimum pixel count for each superpixel, and the last column the the average
AUROC for the 5 tested scenarios.

## References

  * [1] M. E. Paoletti, J. M. Haut, R. Fernandez-Beltran, J. Plaza, A. Plaza, J. Li, and F. Pla, “Capsule networks for hyperspectral image classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 57, no. 4, pp. 2145–2160, 2018.
  * [2] W. D. Ellis, _A source book of Gestalt psychology_. Routledge, 2013.
  * [3] D. Konstantinidis, “Building detection for monitoring of urban changes,” 2017\.
  * [4] C. Almeida, W. Fernandes, S. Barbosa, and H. Lopes, “Dealing with heterogeneous google earth images on building area detection task,” in _Iberoamerican Congress on Pattern Recognition_. Springer, 2018, pp. 133–140.
  * [5] X. Li, X. Yao, and Y. Fang, “Building-a-nets: Robust building extraction from high-resolution remote sensing images with adversarial networks,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 11, no. 10, pp. 3680–3687, 2018.
  * [6] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _CVPR_ , 2015, pp. 3431–3440.
  * [7] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _MICCAI_. Springer, 2015, pp. 234–241.
  * [8] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE TPAMI_ , vol. 39, no. 12, pp. 2481–2495, 2017.
  * [9] S. Minaee, Y. Y. Boykov, F. Porikli, A. J. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” _IEEE TPAMI_ , 2021.
  * [10] I. Ulku and E. Akagündüz, “A survey on deep learning-based architectures for semantic segmentation on 2d images,” _Applied Artificial Intelligence_ , pp. 1–45, 2022.
  * [11] R. Sekkal, C. Strauss, F. Pasteau, M. Babel, and O. Déforges, “Fast pseudo-semantic segmentation for joint region-based hierarchical and multiresolution representation,” in _Visual Information Processing and Communication III_ , vol. 8305. International Society for Optics and Photonics, 2012, p. 83050J.
  * [12] C. Geng, S.-j. Huang, and S. Chen, “Recent advances in open set recognition: A survey,” _IEEE TPAMI_ , vol. 43, no. 10, pp. 3614–3631, 2020.
  * [13] H. Oliveira, C. Silva, G. L. Machado, K. Nogueira, and J. A. dos Santos, “Fully convolutional open set segmentation,” _Machine Learning_ , pp. 1–52, 2021.
  * [14] M. Vendramini, H. Oliveira, A. Machado, and J. A. dos Santos, “Opening deep neural networks with generative models,” in _ICIP_. IEEE, 2021, pp. 1314–1318.
  * [15] C. E. Rasmussen, “Gaussian processes in machine learning,” in _Summer School on Machine Learning_. Springer, 2003, pp. 63–71.
  * [16] C. M. Bishop and N. M. Nasrabadi, _Pattern Recognition and Machine Learning_. Springer, 2006, vol. 4, no. 4.
  * [17] Q. Lin, W. Zhong, and J. Lu, “Deep superpixel cut for unsupervised image segmentation,” in _ICPR_. IEEE, 2021, pp. 8870–8876.
  * [18] P. F. Felzenszwalb and D. P. Huttenlocher, “Efficient graph-based image segmentation,” _International Journal of Computer Vision_ , vol. 59, no. 2, pp. 167–181, 2004.
  * [19] T. Verelst, M. Blaschko, and M. Berman, “Generating superpixels using deep image representations,” _arXiv preprint arXiv:1903.04586_ , 2019.
  * [20] M. Wang, X. Liu, N. Q. Soomro, G. Han, and W. Liu, “Content-sensitive superpixel segmentation via self-organization-map neural network,” _Journal of Visual Communication and Image Representation_ , vol. 63, p. 102572, 2019.
  * [21] P. C. Mahalanobis, “On the generalized distance in statistics.” National Institute of Science of India, 1936.
  * [22] I. Nunes, M. B. Pereira, H. Oliveira, J. A. d. Santos, and M. Poggi, “Conditional reconstruction for open-set semantic segmentation,” _arXiv preprint arXiv:2203.01368_ , 2022.
  * [23] J. Ji, X. Lu, M. Luo, M. Yin, Q. Miao, and X. Liu, “Parallel fully convolutional network for semantic segmentation,” _IEEE Access_ , vol. 9, pp. 673–682, 2020.
  * [24] L. Melas-Kyriazi and A. K. Manrai, “Pixmatch: Unsupervised domain adaptation via pixelwise consistency training,” in _CVPR_ , 2021, pp. 12 435–12 445.
  * [25] J. Kang, Z. Wang, R. Zhu, X. Sun, R. Fernandez-Beltran, and A. Plaza, “Picoco: Pixelwise contrast and consistency learning for semisupervised building footprint segmentation,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 14, pp. 10 548–10 559, 2021.
  * [26] R. Ratajczak, C. Crispim, B. Fervers, E. Faure, and L. Tougne, “Semantic segmentation post-processing with colorized pairwise potentials and deep edges,” in _International Conference on Image Processing Theory, Tools and Applications_. IEEE, 2020, pp. 1–6.
  * [27] Z. Hu, Q. Zou, and Q. Li, “Watershed superpixel,” in _ICIP_. IEEE, 2015, pp. 349–353.
  * [28] S. Xie and Z. Tu, “Holistically-nested edge detection,” in _ICCV_ , 2015, pp. 1395–1403.
  * [29] H. Zhang, K. Jiang, Y. Zhang, Q. Li, C. Xia, and X. Chen, “Discriminative feature learning for video semantic segmentation,” in _International Conference on Virtual Reality and Visualization_. IEEE, 2014, pp. 321–326.
  * [30] W. J. Scheirer, A. de Rezende Rocha, A. Sapkota, and T. E. Boult, “Toward open set recognition,” _IEEE TPAMI_ , vol. 35, no. 7, pp. 1757–1772, 2012.
  * [31] A. Bendale and T. E. Boult, “Towards open set deep networks,” in _CVPR_ , 2016, pp. 1563–1572.
  * [32] C. C. da Silva, K. Nogueira, H. N. Oliveira, and J. A. dos Santos, “Towards open-set semantic segmentation of aerial images,” in _IEEE Latin American GRSS & ISPRS Remote Sensing Conference_. IEEE, 2020, pp. 16–21.
  * [33] S. Prasad, B. Le Saux, N. Yokoya, and R. Hansch, “2018 IEEE GRSS Data Fusion Challenge,” 2020. [Online]. Available: https://dx.doi.org/10.21227/jnh9-nz89
  * [34] J. A. C. Martinez, H. Oliveira, J. A. dos Santos, and R. Q. Feitosa, “Open set semantic segmentation for multitemporal crop recognition,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 19, pp. 1–5, 2021.
  * [35] Z. Cui, W. Longshi, and R. Wang, “Open set semantic segmentation with statistical test and adaptive threshold,” in _IEEE International Conference on Multimedia and Expo_. IEEE, 2020, pp. 1–6.
  * [36] J. Cen, P. Yun, J. Cai, M. Y. Wang, and M. Liu, “Deep metric learning for open world semantic segmentation,” in _ICCV_ , 2021, pp. 15 333–15 342.
  * [37] J. Hong, W. Li, J. Han, J. Zheng, P. Fang, M. Harandi, and L. Petersson, “Goss: Towards generalized open-set semantic segmentation,” _arXiv preprint arXiv:2203.12116_ , 2022.
  * [38] A. Vedaldi and S. Soatto, “Quick shift and kernel methods for mode seeking,” in _ECCV_. Springer, 2008, pp. 705–718.
  * [39] A. Levinshtein, A. Stere, K. N. Kutulakos, D. J. Fleet, S. J. Dickinson, and K. Siddiqi, “Turbopixels: Fast superpixels using geometric flows,” _IEEE TPAMI_ , vol. 31, no. 12, pp. 2290–2297, 2009.
  * [40] M.-Y. Liu, O. Tuzel, S. Ramalingam, and R. Chellappa, “Entropy rate superpixel segmentation,” in _CVPR_. IEEE, 2011, pp. 2097–2104.
  * [41] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “Slic superpixels compared to state-of-the-art superpixel methods,” _IEEE TPAMI_ , vol. 34, no. 11, pp. 2274–2282, 2012.
  * [42] P. Morerio, L. Marcenaro, and C. S. Regazzoni, “A generative superpixel method,” in _International Conference on Information Fusion_. IEEE, 2014, pp. 1–7.
  * [43] P. Buyssens, M. Toutain, A. Elmoataz, and O. Lézoray, “Eikonal-based vertices growing and iterative seeding for efficient graph-based segmentation,” in _ICIP_. IEEE, 2014, pp. 4368–4372.
  * [44] M. V. d. Bergh, X. Boix, G. Roig, B. d. Capitani, and L. V. Gool, “Seeds: Superpixels extracted via energy-driven sampling,” in _ECCV_. Springer, 2012, pp. 13–26.
  * [45] Z. Li and J. Chen, “Superpixel segmentation using linear spectral clustering,” in _CVPR_ , 2015, pp. 1356–1363.
  * [46] V. Machairas, M. Faessel, D. Cárdenas-Peña, T. Chabardes, T. Walter, and E. Decenciere, “Waterpixels,” _IEEE TIP_ , vol. 24, no. 11, pp. 3707–3716, 2015.
  * [47] A. Rubio, L. Yu, E. Simo-Serra, and F. Moreno-Noguer, “Bass: boundary-aware superpixel segmentation,” in _ICPR_. IEEE, 2016, pp. 2824–2829.
  * [48] R. Achanta, P. Márquez-Neila, P. Fua, and S. Süsstrunk, “Scale-adaptive superpixels,” in _Color and Imaging Conference_ , vol. 2018, no. 1. Society for Imaging Science and Technology, 2018, pp. 1–6.
  * [49] J. Zhang, P. Wang, F. Gong, H. Zhu, and N. Chen, “Content-based superpixel segmentation and matching using its region feature descriptors,” _IEICE Transactions on Information and Systems_ , vol. 103, no. 8, pp. 1888–1900, 2020\.
  * [50] E. Elkhateeb, H. Soliman, A. Atwan, M. Elmogy, K.-S. Kwak, and N. Mekky, “A novel coarse-to-fine sea-land segmentation technique based on superpixel fuzzy c-means clustering and modified chan-vese model,” _IEEE Access_ , vol. 9, pp. 53 902–53 919, 2021.
  * [51] S. Lloyd, “Least squares quantization in pcm,” _IEEE Transactions on Information Theory_ , vol. 28, no. 2, pp. 129–137, 1982.
  * [52] J. B. Kruskal, “On the shortest spanning subtree of a graph and the traveling salesman problem,” _Proceedings of the American Mathematical society_ , vol. 7, no. 1, pp. 48–50, 1956.
  * [53] X. Sun, Z. Yang, C. Zhang, K.-V. Ling, and G. Peng, “Conditional gaussian distribution learning for open set recognition,” in _CVPR_ , 2020, pp. 13 480–13 489.
  * [54] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _CVPR_ , 2017, pp. 4700–4708.
  * [55] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in _British Machine Vision Conference_. British Machine Vision Association, 2016.
  * [56] D. Hendrycks, M. Mazeika, and T. Dietterich, “Deep anomaly detection with outlier exposure,” _arXiv preprint arXiv:1812.04606_ , 2018.
  * [57] R. Yoshihashi, W. Shao, R. Kawakami, S. You, M. Iida, and T. Naemura, “Classification-reconstruction learning for open-set recognition,” in _CVPR_ , 2019, pp. 4016–4025.
  * [58] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _NeurIPS_. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [59] J. Cohen, “A coefficient of agreement for nominal scales,” _Educational and Psychological Measurement_ , vol. 20, no. 1, pp. 37–46, 1960.
  * [60] F. Yang, Q. Sun, H. Jin, and Z. Zhou, “Superpixel segmentation with fully convolutional networks,” in _CVPR_ , 2020, pp. 13 964–13 973.
  * [61] J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang _et al._ , “Deep high-resolution representation learning for visual recognition,” _IEEE TPAMI_ , vol. 43, no. 10, pp. 3349–3364, 2020\.

|  Ian Monteiro Nunes currently applying for the PhD degree at the Department
of Informatics at Pontifical Catholic University of Rio de Janeiro (PUC-Rio),
Rio de Janeiro - Brasil, where he also completed the MSc. degree in 2008 and
the BSc. degree in Computer Engeneering in 2003. Research areas include
Machine Learning, Deep Learning, Domain Adaptation, Computer Vision, Remote
Sensing, Clustering and Open Set Recognition.
---|---
|  Matheus Pereira currently pursuing the PhD degree at the Department of
Computer Science at Universidade Federal de Minas Gerais (UFMG), Belo
Horizonte - Brazil, where he also completed the MSc. degree in 2019. Bsc.
(2017) degree in Computer Science by Universidade Federal de Lavras (UFLA).
Research areas include Machine Learning, Computer Vision, Remote Sensing,
Long-Tailed Recognition and Semantic Segmentation.
---|---
|  Hugo Oliveira currently a Post-Doctoral fellow at the Institute of
Mathematics and Statistics at Universidade de São Paulo (IME/USP). BSc. (2014)
and MSc. (2016) degrees in Computer Science by Universidade Federal da Paraíba
(UFPB). PhD in Computer Science (2020) by Universidade Federal de Minas Gerais
(UFMG), Belo Horizonte, Brazil. Research areas include Machine Learning, Deep
Learning, Domain Adaptation, Deep Generative Models, Few-Shot Learning, Meta-
Learning, Metric-Learning, Self-Supervised Learning and Open Set Recognition.
---|---
|  Jefersson Alex dos Santos has got a Ph.D in Computer Science from
Université de Cergy-Pontoise (France) and from University of Campinas (Brazil)
in 2013. Currently, he is an Adjunct Professor in the Department of Computer
Science at the Universidade Federal de Minas Gerais, Brazil. He has a Research
Productivity scholarship from the Brazilian Research Council (CNPq) since
2016. Jefersson has published several articles in journals with high impact
factor and selective editorial policy. He has also published more than thirty
articles in important conferences of remote sensing, image processing and
computer vision areas. Jefersson has experience in coordinating research with
Brazilian funding agencies and R&D projects with companies in those topics.
Jefersson is founder and coordinator of the Laboratory of Pattern Recognition
and Earth Observation (PATREO - www.patreo.dcc.ufmg.br), one of Brazil’s
pioneer groups focused on the development of computer vision and machine
learning for remote sensing applications.
---|---
|  Marcus Poggi is an Associate Professor at PUC-Rio and his main area of
expertise is Computer Science with emphasis on Theory of Computation and
Optimization and Automated Reasoning. In these areas he conducts research in
Industrial Automation, Decision Support Systems, Operational Research, Design
andAnalysis of Algorithm at the Galgos laboratory. He has an undergraduate
degree in Electrical Engineering from Pontifícia Universidade Católica do Rio
de Janeiro (1983), a master’s degree in Electrical Engineering also from PUC-
Rio (1988) and a PhD in Applied Mathematics from Ecole Polytechnique de
Montreal (1993).
---|---