#  Revisiting Image Reconstruction for Semi-supervised Semantic Segmentation

Yuhao Lin, Haiming Xu, Lingqiao Liu, Jinan Zou, Javen Qinfeng Shi
Australian Institute for Machine Learning
University of Adelaide
{yuhao.lin01, hai-ming.xu, lingqiao.liu, jinan.zou<EMAIL_ADDRESS>
###### Abstract

Autoencoding, which aims to reconstruct the input images through a bottleneck
latent representation, is one of the classic feature representation learning
strategies. It has been shown effective as an auxiliary task for semi-
supervised learning but has become less popular as more sophisticated methods
have been proposed in recent years. In this paper, we revisit the idea of
using image reconstruction as the auxiliary task and incorporate it with a
modern semi-supervised semantic segmentation framework. Surprisingly, we
discover that such an old idea in semi-supervised learning can produce results
competitive with state-of-the-art semantic segmentation algorithms. By
visualizing the intermediate layer activations of the image reconstruction
module, we show that the feature map channel could correlate well with the
semantic concept, which explains why joint training with the reconstruction
task is helpful for the segmentation task. Motivated by our observation, we
further proposed a modification to the image reconstruction task, aiming to
further disentangle the object clue from the background patterns. From
experiment evaluation on various datasets, we show that using reconstruction
as auxiliary loss can lead to consistent improvements in various datasets and
methods. The proposed method can further lead to significant improvement in
object-centric segmentation tasks.

##  1 Introduction

Autoencoding aims to reconstruct inputs as outputs with the least possible
amount of distortion[2] through an information bottleneck created with low
dimension or low-resolution latent variables. Because of its simplicity and
effectiveness, it has attracted researchers’ attention since it was first
introduced in the 1980s[37]. Autoencoders again enter the visions of the
researchers when the deep stacked-autoencoder architectures[19] have shown
state-of-the-art results as a feature extractor. Nowadays, as one of the most
classic representation learning strategies, autoencoder has been widely
applied in different applications, such as clustering [17], and classification
[32]. It has also been discovered [26, 15] that an auto-encoder style
reconstruction task could be an excellent auxiliary task for semi-supervised
learning. In semi-supervised learning settings, we have access to a large
volume of unlabelled data and only a small number of labeled training samples.
The reconstruction objective, however, can be applied without any class
labels. However, only a small number of labeled training samples and the
reconstruction loss can be trained without any class labels. Such a scheme has
become less popular as more sophisticated methods [8, 20, 30] have been
proposed recently.

Semi-supervised semantic segmentation is a challenging yet important topic in
computer vision, with many real-world applications. It requires utilizing both
labeled and unlabeled data to improve segmentation results. As an important
application of semi-supervised learning, many semi-supervised learning methods
[38, 22] have been applied and extended to solve the semi-supervised
segmentation problem [46, 41, 20]. However separating the fuzzy margin between
foreground and background is still challenging.

Figure 1: Some visualization results from Pascal VOC 2012 validation set (the
reconstruction branch is only trained on the training set). (a) input images,
(b) feature maps from the reconstruction-only model, (c) feature maps from the
reconstruction-segmentation model, and (d) the segmentation results of the
same epoch as c. We observe that the activation areas in some feature maps are
focused on the objects (i.e., birds in the first row and goats in the fourth
row).

This work explores the use of an autoencoder-style reconstruction task to
improve semi-supervised segmentation. Perhaps surprisingly, we find that if we
incorporate the image reconstruction task with a commonly used semi-supervised
segmentation baseline method [27, 46], the final performance can be improved,
especially when the number of training images is small (see Figure 2). This
motivates us to understand how reconstruction helps the segmentation task. By
visualizing the intermediate activation maps of the reconstruction branch (see
Figure 1), we find that the latent activations of a reconstruction branch have
already uncovered the semantics of objects if the reconstruction branch is
jointly trained with a semi-supervised segmentation loss. This explains the
benefit of the reconstruction task for semi-supervised segmentation, as both
tasks are shared with some similarities. From further observation of the
latent activations, we noticed that the object and part of its background
could often co-occur in one feature map, suggesting potential entanglement of
the object and background clue. Thus, we propose a strategy to further
disentangle those two clues and expect to align the reconstruction task and
segmentation task better. Specifically, we propose to reconstruct foreground-
region-only images for the labeled image and perform a similar reconstruction
task but apply the loss to partial pixels, as guided by the pseudo-label
generated from the segmentation head. Through our experimental evaluation on
various datasets, we show that joint reconstruction can be used as a strong
semi-supervised segmentation baseline that achieves consistent improvement
under different scenarios.

The main contributions of this paper are highlighted as follows:

  * •

We revisit image construction as an auxiliary task for semi-supervised
segmentation and show that it can be very effective when working together with
existing semi-supervised segmentation methods.

  * •

We visualize the intermediate activations of the reconstruction decoder and
shed light on why it is beneficial for the segmentation task.

  * •

We further propose a method that modifies the reconstruction task and makes
the reconstruction tasks more suited for the object-centric segmentation
problems.

##  2 Related Work

###  2.1 Autoencoders for Semi-Supervised Learning

As one classic unsupervised representation learning approach, Autoencoders
(AE) are widely used for unsupervised learning and as a regularization scheme
in semi-supervised learning[2, 23]. Because of the simplicity, they have been
attracting researchers’ attention since it was first introduced in the
1980s[37]. In semi-supervised learning settings, they have a considerable
volume of unlabelled data but only a small number of labeled training samples.
In this setting, some study [35] shows that skip connections and layer-wise
unsupervised targets effectively turn autoencoders into hierarchical latent
variable models, which are well suited for semi-supervised learning. Because
of its nature of clustering, it has been shown effective as an auxiliary task
for many semi-supervised learning tasks such as regression [17], and
classification [32]. Based on the success of AE, denoising autoencoders
(DAE)[40] and variational autoencoders (VAE)[25] are proposed for better
representation learning. However, it has become less popular as more
sophisticated methods have been proposed in recent years.

|
---|---
(a) | (b)
Figure 2: (a) illustrates a simple diagram of our joint training process. The
model is based on the traditional teacher-student structure, and we add
another decoder with the same architecture as the segmentation decoder to
reconstruct the original image. When training, the optimization goal is to
minimize the total loss from both segmentation CE loss and reconstruction MSE
loss. In (b), we compare the performance of baseline with and without original
image reconstruction on Pascal VOC 2012[11] under the standard partition
protocols. It is worth noting that even by simply reconstructing the input
image, the baseline outperforms current SOTA results in all partitions.

###  2.2 Supervised Semantic Segmentation

As the fundamental task in computer vision, semantic segmentation has
witnessed an explosion of progress in architecture design during recent
decades. Starting from the FCN [31], which modifies the end-to-end
architecture to be fully convolutional layers. After that, some extensions
were explored: 1) encoder-decoder structure[1, 6, 36], 2) multi-scale aspects
of the image [5, 29], 3) pyramidal feature maps[47], 4) dilated convolutions
[3, 6, 44]. Recently, attention mechanisms[14, 5, 21] are popular among
researchers because of their strength in global context communication.
However, these fully supervised segmentation networks are data-hungry,
laborious, and time-consuming.

Figure 3: Illustration of our approach. The left side is the basic MT
structure which employs two segmentation networks, the student $h_{s},g_{s}$
and the teacher $h_{t},g_{t}$. The right side $g_{rec}$ is our Foreground-Only
reconstruction module, which takes the output of the student’s encoder $h_{s}$
and reconstructs a monotonous background image. Reconstruction loss masks are
applied for loss calculation. Specifically, for labeled images, we set the
background to 0 according to the ground truth, and the one-hot mask
$M_{rec}^{sup}$ in shape [${H\times W\times 1}$] is all ones. For unlabeled
images, the mask $M_{rec}^{unsup}$ is derived from the teacher’s ignoring
uncertain areas. Then the teacher is updated by the student’s exponential
moving average (EMA).

###  2.3 Semi-supervised Semantic Segmentation

Semi-supervised semantic segmentation aims to utilize the tremendous unlabeled
and small amount of labeled data fully. Though the emergence of attention
mechanisms for the model attracts much attention in the segmentation field,
SOTA semi-supervised semantic segmentation still relies on the CNN model
Deeplab V3+ [4] and PSPNet [47]. One of the most existing studies used
approaches is consistent learning with various perturbations[46, 33, 24, 30].
For example, cutmix-seg [46] validates the effectiveness of image-level
perturbation with cutmix data augmentation. Cross-consistency training
(CCT)[33] introduces a feature-level perturbation and constrains the outputs
of different decoders. Similarly, guided collaborative training (GCT)[24]
proposed a model-level perturbation with different network initialization and
enforced the consistency between models. Most recently, PS-MT[30] introduced a
new adversarial perturbation for double teachers for better prediction
accuracy.

Another frequently used technique is self-training[45, 8, 43], which generates
pseudo labels with unlabeled data and trains the model with labeled and
pseudo-label data. Furthermore, considering the class imbalance and unreliable
pseudo labels, recent studies [20, 16, 41] were proposed and achieved state-
of-the-art performance.

##  3 Image Reconstruction

Autoencoder [19] is one of the oldest unsupervised/self-supervised learning
approaches. It is usually implemented by an encoder-decoder pair. The encoder
encodes the input image information into latent variables that are often low-
dimensional or low-resolution. Then the decoder decodes the latent variables
into an image of the same size as the input. A loss function is used to ensure
the reconstructed output image is as close as possible to the input image,
that is,

$\displaystyle\min_{\theta_{e},\theta_{d}}\
\mathbb{E}\Bigl{(}\|x-f_{\theta_{d}}(f_{\theta_{e}}(x))\|^{2}_{2}\Bigl{)},$
(1)

where $\mathbb{E}$ is a dissimilarity or distortion function and
$f_{\theta_{d}}$, $f_{\theta_{e}}$ are the decoder and encoder, respectively.

Modern image segmentation neural networks, such as the DeepLab family [34, 13,
3] can also be seen as an encoder-decoder structure, although the decoder is
usually lightweight compared to the decoder. The encoder encodes the images
into a $H^{\prime}\times W^{\prime}\times d$ dimensional feature map, and the
decoder, e.g., ASPP module decodes the feature maps into the predicted
segmentation mask. Therefore, the image reconstruction task can be naturally
applied to the existing image segmentation neural networks by sharing the same
encoder but using a different decoder (branch) to produce reconstructed
images.

##  4 Image Reconstruction for Semi-supervised Semantic Segmentation

###  4.1 Preliminary

Semi-supervised semantic segmentation task is defined as: given labeled images
$X_{l}\in\mathbb{R}^{H\times W\times 3}$ , the corresponding pixel-wise
semantic labels $y\in(1,C)^{H\times W}$, and unlabelled images
$X_{u}\in\mathbb{R}^{H\times W\times 3}$ (W, H, C denotes the width, height,
and the number of classes respectively). The goal is to learn a model $F$ from
both label data $D^{l}=\\{X_{l},y\\}$ and unlabelled data $D^{u}=\\{X_{u}\\}$.
In most work [46, 33, 24, 30, 41], the overall optimization target is
formalized as

$\mathcal{L}=\mathcal{L}_{s}+\lambda\mathcal{L}_{ul},$

where $\mathcal{L}_{s}$ and $\mathcal{L}_{ul}$ are loss functions for labeled
and unlabeled images respectively.

###  4.2 Semi-supervised Segmentation Baseline

Most of the state-of-the-art semi-supervised semantic segmentation methods
[45, 8, 43] are built around a simple baseline, as we call robust pseudo-
labeling. Specifically, the model is firstly trained on the small number of
labeled images and then produces the posterior probability estimation for each
pixel on unlabeled images. Pseudo-labels are then generated if the highest
posterior probability exceeds a predefined threshold. In robust pseudo-
labeling, certain data augmentation, e.g., cutout [10], cutmix [46], is
applied to the input image, and the pseudo-labels will be used to update the
model with the augmented input images.

Our semi-supervised segmentation baseline is based on a particular version of
the robust pseudo-labeling approach. Specifically, our baseline follows the
typical student-teacher framework in semi-supervised semantic segmentation
[33, 24, 30, 41], with the teaching network parameters being the exponential
moving average [39] of the parameters of the student network.

Each network consists of a convolutional feature encoder $h$ and a
segmentation decoder $g$. We denote the student version and teacher version of
the encoder and decoder as $h_{s}$,$g_{s}$, $h_{t}$ $g_{t}$, respectively. At
each training step, we equally sample $b$ labeled images $\mathcal{B}_{l}$ and
$b$ unlabeled images $\mathcal{B}_{ul}$. For $\mathcal{B}_{l}$ and
$\mathcal{B}_{ul}$, we apply strong augmentations [8, 24] (e.g., color jitter,
randomize grayscale, blur, CutMix [46] and zoom in/out [28, 5]) to the student
model. The teacher model will generate posterior distribution
$P(y_{i,j}=c|I^{n}_{i,j})$, indicating the likelihood of each pixel $(i,j)$
being assigned to class $c$. A pseudo-label for pixel $(i,j)$ is generated if
$\max_{c}P(y_{i,j}=c|I^{n}_{i,j})>\tau$. Then the pseudo-labels will be used
to train a student network with augmented input images.

###  4.3 Reconstruction as an Auxiliary Task

Based on the framework mentioned in the Section 4.2, we further incorporate
image reconstruction as an auxiliary task, which is shown in Figure
LABEL:toymodel. Different from the baseline, our framework has two decoders
($g_{s}$,$g_{rec}$) in the student network, and the two decoders share the
encoder part ($h_{s}$).

Therefore, the outputs of the model are segmentation prediction:
$P_{seg}=g_{s}\circ h_{s}(x)\in\mathbb{R}^{H\times W\times C}$ and the image
reconstruction pixel value prediction $I_{rec}=g_{rec}\circ
h_{s}(x)\in\mathbb{R}^{H\times W\times 3}$. Same as [7], we adopt the Mean
Squared Error (MSE) loss for image reconstruction, and the image
reconstruction module does not need additional annotations. The overall loss
is defined as

$\mathcal{L}=\mathcal{L}_{s}+\lambda_{1}\mathcal{L}_{ul}+\lambda_{2}\mathcal{L}_{rec}.$

Surprisingly, this embarrassingly simple baseline achieves quite good
performance. Figure 2 shows the performance before and after adding the
reconstruction task. As seen, the benefit of using a reconstruction task is
especially pronounced when the number of labeled images is small.

####  4.3.1 Visualization the “Latent Images” from the Reconstruction Decoder

To understand the improvement, we perform visualization analysis on the
reconstruction decoder. In particular, we consider the latent activations
(feature maps) before the last layer of the reconstruction decoder. Recall
that this last layer is a (kernel size $=1\times 1$) convolutional layer,
which maps a $\mathbf{Z}\in\mathbf{R}^{H\times W\times d}$ feature map into
the reconstructed image $I_{rec}\in\mathbb{R}^{H\times W\times 3}$. There are
three convolutional filters
$\mathbf{w}_{1},\mathbf{w}_{2},\mathbf{w}_{3}\in\mathbb{R}^{d}$, one for each
color channel. Now consider one output channel from $I_{rec}$, denoted as
$I^{c}_{rec}$, it is clear that it can be written as

$\displaystyle I^{c}_{rec}=\sum_{k=1}^{d}\mathbf{Z}^{k}w^{k}_{c},$ (2)

where $w^{k}_{c}$ denotes the $k$-th dimension of $\mathbf{w}_{c}$ and
$\mathbf{Z}^{k}\in\mathbb{R}^{H\times W}$ denotes the $k$-th slice of
$\mathbf{Z}$. Intuitively, the above equation suggests that the reconstructed
image is the weighted average of $d$ slices $\mathbf{Z}^{k}$, where each
$\mathbf{Z}^{k}$ is equivalent to an image, and we call it “latent image”.

An interesting discovery is that those “latent images” could correspond to the
semantic concepts in images if the encoder is jointly trained with a
segmentation model. Figure 1 shows some example “latent images” from an
autoencoder trained with reconstruction loss only and trained jointly with
semi-supervised segmentation loss. Also, to show the relative progress of the
segmentation decoder and the reconstruction decoder, we choose an epoch that
the training process has not converged yet. From Figure 1, we can make the
following observations:

  * •

If the autoencoder is trained without the semi-supervised segmentation loss,
the “Latent Images” have a weak correlation to the semantic concepts.

  * •

When the autoencoder is trained with the semi-supervised segmentation loss,
some “Latent Images” can correspond to some semantic concepts. For example, in
the second and fourth row of Figure 1, the activation areas in some feature
maps are focused on the train and the sheep. It seems that the semantic
segmentation loss provides an inductive bias to make the image reconstructed
through semantically meaningful “latent images”.

  * •

Surprisingly, some “Latent Images” recover the object contour better than the
segmentation decoder at the same training epoch. It seems that the “Latent
Images” are leading the segmentation decoder, which might explain why the
reconstruction task could help segmentation.

  * •

Finally, we find the “Latent Images” are far from perfect. Some background
pixels, especially those that are the context of the object, tend to co-occur
with the object in the latent image.

###  4.4 Improving the Reconstruction Task by Object-Background
Disentanglement

The last observation discussed in Section 4.3.1 suggests a potential object-
background entanglement may exist in the current reconstruction task. Thus, we
propose the following strategy to disentangle the object and its context
background. More specifically, we let the reconstruction decoder only
reconstruct the foreground-only images at the labeled set. In other words, the
output from the reconstruction decoder only contains pixels belonging to the
object parts while the background pixels are set to zero, that is,

$y_{rec}\ {}_{(i,j)}=\begin{cases}x^{l}_{(i,j)},&\text{if}\
y_{(i,j)}\in\text{foreground}\\\ 0,&\text{otherwise}\end{cases}$ (3)

Examples of foreground-only images are shown in Figure 3. For unlabeled
images, we do not have access to the class labels, those we cannot directly
generate foreground-only images. Thus we recourse to pseudo-labels. For an
unlabeled image, we consider the following three scenarios for a pixel
$(i,j)$: (1) the current pixel can generate a pseudo-label, and the pseudo-
label corresponds to the foreground. In other words,
$max_{c^{\prime}\in\mathcal{O}}P(y_{i,j}=c^{\prime}|x_{i,j})>\tau$, where
$\mathcal{O}$ is the set of classes that belong to objects. (2) the current
pixel can generate a pseudo-label, and the pseudo-label corresponds to the
background. (3) no pseudo-label can be generated from the current pixel and
the segmentation decoder is uncertain about the class of the pixel. We will
ignore the loss of pixels from (3) and only perform the foreground-only
reconstruction for pixels from (1) and (2).

We call this modified reconstruction method as Foreground-Only reconstruction
(FOrec). The scheme is illustrated in Figure 3.

Discussion: The FOrec method is mainly for object centric semantic
segmentation, where the aim is to segment different objects from the
background. For generic scene segmentation, i.e., segmenting both things and
stuff, one can choose a category that often co-occurs with other categories as
the background. Applying FOrec in that case could potentially alleviate the
entanglement of those semantic concepts.

##  5 Experiments

In this section, we compare our approach with several semi-supervised semantic
segmentation methods.

###  5.1 Experimental Setup

Datasets: Our experiments are mainly conducted on the Pascal VOC 2012 [11],
and Cityscapes [9], which are widely used in semi-supervised semantic
segmentation tasks [46, 33, 24, 30]. The classic Pascal VOC 2012 consists of
1,464/1,449/1,556 images covering twenty classes for training, validation, and
testing, respectively. Due to the demand for data for the semi-supervised
semantic segmentation scene, some researchers [33, 24, 30] adapt the
additional labels from [18], which means the training data is augmented up to
10,528 images. In the augmented training set, 1,464 labeled data is selected
among 1,464 samples in the classic setting, while the remainings are of low
quality, containing noise. The augmented data selection setting is named
blender. Both settings are evaluated for the measurement of performance.
Cityscapes [9] is the urban driving scene dataset, consisting of 2,975, 500,
and 1,525 images covering 19 classes for training, validation, and testing,
respectively.

In this paper, we follow the same data splitting protocol from U2PL[41] and
experiment with four kinds of label partition: 1/16, 1/8, 1/4, and 1/2. Our
code will be released after the anonymity period.

Evaluation metrics: Following the previous works [41, 30], we adopt the mean
Intersection-over-Union (mIoU) as the evaluation metrics.

Implement detail: Following the prior work [30, 42, 8], the network structure
of our method is based on Deeplab V3+ [4], with pretrained ResNet-101 as the
backbone. The segmentation head and the auxiliary task head are the default
pixel-level linear classifier.

For all experiments on both datasets, we employ the stochastic gradient
descent (SGD) as the optimizer and polynomial learning rate decay:
$(1-\frac{iter}{total_{i}ter})^{0.8}$ for model optimization. For
reconstruction loss, we set the $\lambda_{1}$ as 0.5 and $\lambda_{2}$ as 1
for the unsupervised and supervised parts, respectively.

On Pascal VOC 2012 [11], the images are cropped into $512\times 512$ pixels
and trained with initial learning rate $1.0\times 10^{-3}$ , weight decay
$1.0\times 10^{-4}$ ,and 80 training epochs. On Cityscapes [9] , we crop the
images into $712\times 712$ pixels and trained our model with an initial
learning rate $1.0\times 10^{-2}$ , weight decay $5.0\times 10^{-4}$ and 200
training epochs. Our experiments were run with batch size 16 on 8 NVIDIA Tesla
V100 GPUs.

For the reconstruction decoder, we simply adapt the same structure as the
segmentation decoder. The only difference between them is the channel number
of the last layer.

###  5.2 Comparison with State-of-the-Arts

####  5.2.1 Pascal VOC 2012

####  5.2.2 Pascal VOC 2012

Table 1 and Table 2 illustrated the results on Pascal VOC 2012 validation set,
Table 1 is under classic setting and Table 2 is under blender setting.

For classic setting, Table 1 illustrates that our approach successfully
exploits unlabelled data, with a dramatic performance boost from the fully
supervised training. Specifically, in the smaller partition like 92 and 183,
our approach surpasses the fully supervised baseline with 25.2% and 19.8%,
respectively. Meanwhile, compared with the current SOTA methods, our approach
performs consistently better than all other methods for all partition
protocols (using the ResNet-101 as the backbone). Taking the U2PL[41] as the
instance, our approach improves the performance by 1.6% to 5.6% in all cases.
Furthermore, in some partitions, our approach is better than all the current
SOTA methods with fewer labeled samples. For example, compared with all
current SOTA methods, our approach trained with 92 labeled images outperforms
trained with 183 labeled images. This demonstrates that when the number of
labeled data is extremely small (92, 183), our approach achieves a significant
improvement in performance.

Method | 92 | 183 | 366 | 732 | 1464
---|---|---|---|---|---
Supervised | 45.8 | 54.9 | 65.9 | 71.7 | 72.5
MT [39] | 51.7 | 58.9 | 63.9 | 69.5 | 71.0
PseudoSeg [48] | 57.6 | 65.5 | 69.1 | 72.4 | 73.2
CPS [8][CVPR 21’] | 64.1 | 67.4 | 71.7 | 75.9 | -
PS-MT[30][CVPR 22’] | 65.8 | 69.6 | 76.6 | 78.4 | 80.0
ST++[43][CVPR 22’] | 65.2 | 71.0 | 74.6 | 77.3 | 79.1
U2PL[41][CVPR 22’] | 68.0 | 69.1 | 73.6 | 76.2 | 79.5
FOrec (Ours) | 71.0 | 74.7 | 77.5 | 78.7 | 81.1
Table 1: Comparing results of state-of-the-art algorithms on PASCAL VOC 2012
[11] val set with mIoU (%) metric. Methods are trained on the classic setting,
i.e., the labeled images are selected from the original VOC train set, which
consists of 1, 464 samples in total. Best results are in bold.

For blender setting, Table 2 indicates that our approach outperforms the
supervised baseline by a large gap 3.86% to 10.97% from the fully supervised
training. Compared with the current SOTA methods, our approach beats all other
methods for all partition protocols. Even comparing with one of the currently
best performed approaches U2PL[41], our approach improves the performance by
1.63%, 1.51%, 1.62% in 662, 1323, and 2646, respectively. The impressive boost
proves that our method is not only useful for accurately annotated data, but
also compatible with the noisy annotation.

Method | 662 | 1323 | 2646 | 5290
---|---|---|---|---
Supervised | 67.87 | 71.55 | 75.80 | 77.13
MT[39] | 70.51 | 71.53 | 73.02 | 76.58
CPS [8][CVPR 21’] | 74.48 | 76.44 | 77.68 | 78.64
AEL[20] [NIPS 21’] | 77.20 | 77.57 | 78.06 | 80.29
ST++ [43][CVPR 22’] | 74.70 | 77.90 | 77.90 | -
PS-MT [30][CVPR 22’] | 75.50 | 78.20 | 78.72 | 79.76
UCC [12][CVPR 22’] | 76.49 | 77.06 | 79.09 | 79.54
U2PL[41] [CVPR 22’] | 77.21 | 79.01 | 79.30 | 80.50
FOrec (Ours) | 78.84 | 80.52 | 80.92 | 80.99
Table 2: Comparing results of state-of-the-art algorithms on PASCAL VOC 2012
[11] val set with mIoU (%) metric. Methods are trained on the blender setting,
i.e., the labeled images are selected from the augmented VOC train set, which
consists of 10, 582 samples in total. Best results are in bold.

####  5.2.3 Cityscapes

Table 3 demonstrates the results of our method against several current state-
of-the-art algorithms on Cityscapes validation set. Compared to the fully
supervised results, our method successfully exploits unlabelled data, with an
obvious performance boost for all partitions. e.g., under the 1/16 label
partition, our approach surpasses the fully supervised result by 6.68%. Then,
compared to the state-of-the-art algorithm U2PL[41], Ours performs better than
U2PL in all cases by 2.12%, 1.39% and 1.18% under the 1/16, 1/8 and 1/4 label
partition, respectively.

Note that the performance of our method on the 1/16 label partition is
slightly lower than that of AEL [20]. The reason is that the class imbalance
problem of this partition is more serious and AEL especially aims to deal with
class imbalance problems. However, our methods focus on separating the
foreground objects from the background patterns in semi-supervised semantic
segmentation tasks, we do not explicitly consider processing label imbalance
problems. Technically, there is a high probability that merging both ideas is
a to optimize overall performance.

Method | 1/16 | 1/8 | 1/4 | 1/2
---|---|---|---|---
Supervised | 65.74 | 72.53 | 74.43 | 77.83
MT[39] | 69.03 | 72.06 | 74.20 | 78.15
CCT [33][CVPR 20’] | 69.32 | 74.12 | 75.99 | 78.10
CPS [8][CVPR 21’] | 69.78 | 74.31 | 74.58 | 76.81
AEL [20][NIPS 21’] | 74.45 | 75.55 | 77.48 | 79.01
U2PL[41] [CVPR 22’] | 70.30 | 74.37 | 76.47 | 79.05
FOrec (Ours) | 72.42 | 75.76 | 77.65 | 79.18
Table 3: Comparing results of state-of-the-art algorithms on Cityscapes [9] val set with mIoU (%) $\uparrow$ metric. Methods are trained on identical label partitions, and the labeled images are selected from the Cityscapes train set, which consists of 2, 975 samples in total. Best results are in bold. |  Standard Reconstruction |  FOrec | 1/16 (92) | 1/8 (183)
---|---|---|---|---
① | ✗ | ✗ | 67.82 | 70.78
② | ✓ | ✗ | 68.99 | 72.35
③ | ✗ | ✓ | 70.99 | 74.67
Table 4: Ablation study on the effectiveness of different components of our approach. ✓and ✗ represent the variant containing or not containing the sub module at each row respectively.  Method | 92 | 183 | 366 | 732
---|---|---|---|---
fore/back-ground seg. | 67.7 | 71.9 | 76.1 | 78.1
FOrec (ours) | 71.0 | 74.7 | 77.5 | 78.7
Table 5: Foreground-background (saliency estimation) segmentation on PASCAL
VOC 2012 (classic setting).

###  5.3 Analysis

In the following part, we perform a series of experiments to analyze the
proposed method. Specially, we consider the following analyses: (1) The
comparison between FOrec and standard reconstruction on both the object-
centric segmentation task, i.e., PASCAL VOC, and scene-understanding
segmentation task, i.e., CityScapes. (2) The latent images created from FOrec.
(3) The applicability of the proposed method to other semi-supervised
segmentation approaches.

The comparison of the standard reconstruction and FOrec In Table 4, we compare
the standard reconstruction task and the proposed foreground-only
reconstruction scheme. As seen, FOrec achieves a significant improvement over
the standard reconstruction task on PASCAL VOC 2012 clean setting, the
improvement over the standard reconstruction is around 2%. This supports our
claim that using foreground-only images as the reconstruction target could be
beneficial for object-centric segmentation.

The comparison of the foreground-background segmentation and FOrec Table 5
shows that the naive foreground-background segmentation performs worse than
our approach, the improvement benefit from the foreground-only reconstruction
is ranging from 0.5% to 3.3%. We think that unifying all kinds of foregrounds
into one class is not conducive to the semantic segmentation of different
objects.

Figure 4: Visualizations of the latent images obtained from FOrec and standard
reconstruction, from left to right, are (a) input images, (b) latent images
obtained from standard reconstruction, (c) latent images obtained from FOrec,
(d) segmentation results from standard reconstruction, (e) segmentation
results from FOrec, and (f) segmentation ground truth. Note that the (b) and
(c) are slices from the same position for a fair comparison.

The impact of FOrec on the “latent images” The FOrec is proposed to address
the issue that background pixels tend to co-occur with foreground pixels. To
verify this design, we visualize the latent images obtained from FOrec and
standard reconstruction. We conduct an experiment by using two models, one
trained by FOrec and another trained by standard reconstruction (jointly
trained with semi-supervised segmentation loss for both cases with the same
architecture). Then we use the trained model to generate latent images for
input images from the validation set. Note that although FOrec is trained to
reconstruct foreground-only images, the Foreground-only images are only used
at the training time as the target. Once trained, it can be used without
knowing the foreground mask. The results are shown in Figure 4. As seen, by
applying FOrec, the latent images tend to capture more object regions. We can
observe more “latent images” that only include the objects. For example, in
our figures, the goats and the cat in (c) are highlighted a lot compared with
the latent images of standard reconstruction. When mapping to the results of
the segmentation task, these natures are kept. For instance, in the first
example in Figure 4, the gaps between goats’ legs are captured, which did not
appear in the results from the standard reconstruction. Moreover, in the
second example, the sofa next to the cat is faded in the FOrec feature maps.
In the segmentation results, it disappears. These results in Figure 4 clearly
validate the effectiveness of the proposed method.

Method | 92 | 183 | 366 | 732
---|---|---|---|---
PS-MT[30] | 65.8 | 69.6 | 76.6 | 78.4
PS-MT+rec | 68.4 | 71.0 | 77.2 | 78.9
PS-MT+FOrec | 70.3 | 71.9 | 77.9 | 79.8
Table 6: Comparing results of another SOTA codebase (PS-MT[30]) on classic
setting PASCAL VOC 2012 [11] val set with mIoU (%) metric. Experiments are
conducted in the same settings as in the paper.

The applicability of the proposed method on other semi-supervised segmentation
methods Finally, we apply both the reconstruction task and FOrec to another
semi-supervised learning approach, PS-MT [30]. The results are shown in Table
6. As seen, the reconstruction is still effective. Compared with the PS-MT
baseline, applying FOrec leads to a significant increase, especially when the
number of training examples is small. The advantage of FOrec over standard
reconstruction is also seen. Again, we observe FOrec tends to produce a
superior performance on the low-supervision regime, e.g., when only 92 labeled
images are used.

##  6 Conclusion

In this paper, we revisit the idea of using image reconstruction as an
auxiliary task for semi-supervised semantic segmentation. We find that this
old idea can produce results competitive with state-of-the-art semantic
segmentation algorithms. By visualizing the intermediate layer activations of
the image reconstruction module, we show that the feature map channel can
correlate well with the semantic concept, which explains why joint training
with the reconstruction task is helpful for the segmentation task. Motivated
by this observation, we further proposed a modification to the image
reconstruction task, aiming to further disentangle the object clue from the
background patterns. From experiment evaluation on various datasets, we show
that using reconstruction as an auxiliary loss leads to consistent
improvements in various datasets and methods. The proposed method can further
lead to significant improvement in object-centric segmentation tasks. For
datasets without background class, it only provides slight improvements and
more investigations will be made to improve scene understanding ability.

## References

  * [1] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence, 39(12):2481–2495, 2017.
  * [2] Pierre Baldi.  Autoencoders, unsupervised learning, and deep architectures.  In Proceedings of ICML workshop on unsupervised and transfer learning, pages 37–49. JMLR Workshop and Conference Proceedings, 2012.
  * [3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [4] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [5] Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille.  Attention to scale: Scale-aware semantic image segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3640–3649, 2016.
  * [6] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [7] Shuai Chen, Gerda Bortsova, Antonio García-Uceda Juárez, Gijs van Tulder, and Marleen de Bruijne.  Multi-task attention-based semi-supervised learning for medical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 457–465. Springer, 2019.
  * [8] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang.  Semi-supervised semantic segmentation with cross pseudo supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2613–2622, 2021.
  * [9] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [10] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * [11] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  International journal of computer vision, 111(1):98–136, 2015.
  * [12] Jiashuo Fan, Bin Gao, Huan Jin, and Lihui Jiang.  Ucc: Uncertainty guided cross-head co-training for semi-supervised semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9947–9956, 2022.
  * [13] L-CCGP Florian and Schroff Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  In Conference on computer vision and pattern recognition (CVPR). IEEE/CVF, volume 6, 2017.
  * [14] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3146–3154, 2019.
  * [15] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li.  Deep reconstruction-classification networks for unsupervised domain adaptation.  In European conference on computer vision, pages 597–613. Springer, 2016.
  * [16] Dayan Guan, Jiaxing Huang, Aoran Xiao, and Shijian Lu.  Unbiased subclass regularization for semi-supervised semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9968–9978, 2022.
  * [17] Xifeng Guo, Xinwang Liu, En Zhu, and Jianping Yin.  Deep clustering with convolutional autoencoders.  In International conference on neural information processing, pages 373–382. Springer, 2017.
  * [18] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik.  Semantic contours from inverse detectors.  In 2011 International Conference on Computer Vision, pages 991–998. IEEE, 2011.
  * [19] Geoffrey E Hinton and Ruslan R Salakhutdinov.  Reducing the dimensionality of data with neural networks.  science, 313(5786):504–507, 2006.
  * [20] Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang.  Semi-supervised semantic segmentation via adaptive equalization learning.  Advances in Neural Information Processing Systems, 34:22106–22118, 2021.
  * [21] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu.  Ccnet: Criss-cross attention for semantic segmentation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 603–612, 2019.
  * [22] Minsung Hyun, Jisoo Jeong, and Nojun Kwak.  Class-imbalanced semi-supervised learning.  arXiv preprint arXiv:2002.06815, 2020.
  * [23] Mingeun Kang, Kiwon Lee, Yong H Lee, and Changho Suh.  Autoencoder-based graph construction for semi-supervised learning.  In European Conference on Computer Vision, pages 500–517. Springer, 2020.
  * [24] Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson WH Lau.  Guided collaborative training for pixel-wise semi-supervised learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16, pages 429–445. Springer, 2020.
  * [25] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [26] Lei Le, Andrew Patterson, and Martha White.  Supervised autoencoders: Improving generalization performance with unsupervised regularizers.  Advances in neural information processing systems, 31, 2018.
  * [27] Dong-Hyun Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In Workshop on challenges in representation learning, ICML, volume 3, page 896, 2013.
  * [28] Di Lin, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or, and Hui Huang.  Multi-scale context intertwining for semantic segmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 603–619, 2018.
  * [29] Guosheng Lin, Chunhua Shen, Anton Van Den Hengel, and Ian Reid.  Efficient piecewise training of deep structured models for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3194–3203, 2016.
  * [30] Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, and Gustavo Carneiro.  Perturbed and strict mean teachers for semi-supervised semantic segmentation.  arXiv preprint arXiv:2111.12903, 2021.
  * [31] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [32] Wei Luo, Jun Li, Jian Yang, Wei Xu, and Jian Zhang.  Convolutional sparse autoencoders for image classification.  IEEE transactions on neural networks and learning systems, 29(7):3289–3294, 2017.
  * [33] Yassine Ouali, Céline Hudelot, and Myriam Tami.  Semi-supervised semantic segmentation with cross-consistency training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12674–12684, 2020.
  * [34] George Papandreou, Liang-Chieh Chen, Kevin Murphy, and Alan L Yuille.  Weakly- and semi-supervised learning of a dcnn for semantic image segmentation.  arxiv:1502.02734, 2015.
  * [35] Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko.  Semi-supervised learning with ladder networks.  Advances in neural information processing systems, 28, 2015.
  * [36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [37] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning internal representations by error propagation.  Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
  * [38] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  arXiv preprint arXiv:2001.07685, 2020.
  * [39] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  Advances in neural information processing systems, 30, 2017.
  * [40] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  Journal of machine learning research, 11(12), 2010.
  * [41] Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le.  Semi-supervised semantic segmentation using unreliable pseudo-labels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4248–4257, 2022.
  * [42] Hai-Ming Xu, Lingqiao Liu, Qiuchen Bian, and Zhen Yang.  Semi-supervised semantic segmentation with prototype-based consistency regularization.  arXiv preprint arXiv:2210.04388, 2022.
  * [43] Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao.  St++: Make self-training work better for semi-supervised semantic segmentation.  In CVPR, 2022.
  * [44] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122, 2015.
  * [45] Jianlong Yuan, Yifan Liu, Chunhua Shen, Zhibin Wang, and Hao Li.  A simple baseline for semi-supervised semantic segmentation with strong data augmentation.  arXiv preprint arXiv:2104.07256, 2021.
  * [46] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023–6032, 2019.
  * [47] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  * [48] Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister.  Pseudoseg: Designing pseudo labels for semantic segmentation.  arXiv preprint arXiv:2010.09713, 2020.