# Weakly-Supervised Image Semantic Segmentation Using Graph Convolutional
Networks

###### Abstract

This work addresses weakly-supervised image semantic segmentation based on
image-level class labels. One common approach to this task is to propagate the
activation scores of Class Activation Maps (CAMs) using a random-walk
mechanism in order to arrive at complete pseudo labels for training a semantic
segmentation network in a fully-supervised manner. However, the feed-forward
nature of the random walk imposes no regularization on the quality of the
resulting complete pseudo labels. To overcome this issue, we propose a Graph
Convolutional Network (GCN)-based feature propagation framework. We formulate
the generation of complete pseudo labels as a semi-supervised learning task
and learn a 2-layer GCN separately for every training image by back-
propagating a Laplacian and an entropy regularization loss. Experimental
results on the PASCAL VOC 2012 dataset confirm the superiority of our scheme
to several state-of-the-art baselines. Our code is available at
https://github.com/Xavier-Pan/WSGCN.

Index Terms—  Weakly-supervised image semantic segmentation, Graph
Convolutional Networks

##  1 Introduction

Image semantic segmentation aims for classifying pixels in an image into their
semantic classes. Training a semantic segmentation network often requires
costly pixel-wise class labels. To avoid the time-consuming annotation
process, weakly-supervised learning is introduced to utilize relatively low-
cost weak labels such as bounding boxes, scribbles, and image-level class
labels [1, 2, 3, 4, 5, 6, 7].

Recent research has been focused on weakly-supervised image semantic
segmentation based on image-level class labels. The work in [3] is
representative of the popular two-stage framework. It begins with producing
pseudo labels, followed by training a semantic segmentation network
supervisedly using these labels. The generation of pseudo labels usually
proceeds in three steps. The first step predicts crude estimates of labels,
known as partial pseudo labels, by using Class Activation Maps (CAMs) [8].
They are partial in that only some pixels will receive their pseudo labels. In
the second step, these partial pseudo labels are utilized to train an affinity
network, requiring that pixels sharing the same label should have similar
features. Lastly, the affinity network is applied to evaluate a Markov
transition matrix for propagating the activation scores of CAMs through a
random-walk mechanism, with the aim of producing complete pseudo labels for
all the pixels. On the basis of this two-stage framework, some works [5, 6]
improve on initial CAMs while others [3, 4] attempt to learn better affinity
matrices.

This paper addresses the propagation mechanism through learning a graph neural
network. To derive complete pseudo labels, all the prior works [3, 4, 5, 6]
rely on the random-walk mechanism to propagate the activation scores of CAMs
as a form of label propagation. Due to its feed-forward nature, there is no
regularization imposed on the resulting complete pseudo labels, the quality of
which depends highly on the Markov matrix. Departing from label propagation,
we propose a feature propagation framework based on learning a Graph
Convolutional Network (GCN) [9] for every training image. Our contributions
include:

  1. 1.

We propagate the high-level semantic features of image pixels on a graph using
a 2-layer graph convolutional network, followed by decoding the propagated
features into semantic predictions.

  2. 2.

We cast the problem of obtaining complete pseudo labels from partial pseudo
labels as a semi-supervised learning problem.

  3. 3.

We learn a separate GCN for every training image by back-propagating a
Laplacian loss and an entropy loss to ensure the consistency of semantic
predictions with image spatial details.

Experimental results show that our complete pseudo labels have higher accuracy
in Mean Intersection over Union (mIoU) than label propagation [3, 4]. The net
effect is that the semantic segmentation network trained with our complete
pseudo labels outperforms the state-of-the-art baselines [5, 6, 7, 10, 11, 12]
on the PASCAL VOC 2012 dataset.

Fig. 1: Overview of the proposed 2-stage weakly-supervised image semantic
segmentation framework with GCN-based feature propagation. In our
implementation, the Affinity Network is from [3, 4].

##  2 Related Work

This section surveys prior works that use image-level class labels for weakly-
supervised semantic segmentation, with a particular focus on the improvement
of CAMs and how pseudo labels are derived from CAMs.

###  2.1 Class Activation Maps

The CAMs [8] derived from image-level class labels often serve as initial
seeds for generating partial pseudo labels. Many efforts have been devoted to
improving the quality of CAMs. RRM [10] learns jointly an image classifier and
a semantic segmentation network, with the hope of regularizing the feature
extraction towards better CAM generation. By the same token, SEAM [5] imposes
an equivariant constraint on learning the feature extractor, requiring that
the CAMs of two input images related through an affine transformation follow
the same affine relationship. SC-CAM [6] divides the object category into sub-
categories and learns a feature extractor that can identify find-grained parts
of an object. GroupWSSS [12] learns a graph neural network to leverage the
semantic relations among images which share class labels partially for better
CAM generation.

###  2.2 Generation of Complete Pseudo Labels

There are approaches [2, 3, 4] that focus on generating complete pseudo labels
from CAMs. DSRG [2] expands the partial pseudo labels by annotating
iteratively the unlabeled pixels adjacent to the labeled ones through a
semantic segmentation network. The propagation process, however, does not take
into account the affinity between pixels. By contrast, PSA [3] learns an
affinity network to guide the propagation of the CAM scores by weighting
differently the edges connecting adjacent pixels. The affinity network is
trained by minimizing the $\l_{1}$ norm between the feature vectors of
neighboring pixels that share the same semantic class according to their
partial pseudo labels. IRNet [4] extends the idea to incorporate the boundary
map in determining the affinity between pixels. Specifically, pixels separated
by a strong boundary are considered semantically dissimilar, and vice versa.
SEAM [5] and SC-CAM [6] adopt the same propagation framework for CAMs as [3,
4] to produce complete pseudo labels.

In common, most of these methods build the label propagation on a random-walk
mechanism. The feed-forward nature of the random walk does not provide any
guarantee on the quality of the complete pseudo labels and does not consider
low-level features during propagation. Our work presents the first attempt at
replacing the random walk with a GCN-based feature propagation scheme.

##  3 Proposed Method

This work introduces a GCN-based feature propagation scheme, with the aim of
predicting the semantic label for every pixel to produce complete pseudo
labels. Unlike the random walk, which relies solely on the affinity between
pixels in the feature domain to propagate the activation scores of CAMs, our
scheme learns a GCN by regularizing the feature propagation with not only the
aforementioned affinity information but also the color information of the
input image. Furthermore, recognizing that the generation of pseudo labels is
an off-line process, we train a separate GCN to optimize feature propagation
for every training image. We choose GCN instead of the convolutional neural
networks because of the irregular affinity relations among feature samples.

###  3.1 System Overview

Fig. 1 illustrates the overall architecture of the proposed method. As shown,
the process proceeds in two stages: (1) the generation of pseudo labels, and
(2) the training of a semantic segmentation network. In the first stage, we
follow [3, 10] to generate partial pseudo labels $P\in\mathbb{N}^{H/S\times
W/S}$ for an input image $I\in\mathbb{R}^{3\times H\times W}$ of width $W$ and
height $H$. Note that $P$ is the same size as the CAMs with $S$ denoting the
downscaling factor. At the location $(x,y)$, the pseudo label $P(x,y)$ is
assigned either a class label $c\in C\cup\\{c^{bg}\\}$ or $ignored$, where $C$
is the set of foreground classes, $c^{bg}$ denotes the background class, and
the $ignored$ signals the unlabeled state. Given the partial pseudo labels
$P$, we consider the generation of complete pseudo labels
$\hat{P}\in\mathbb{N}^{H\times W}$ to be a semi-supervised learning problem on
a graph. The output of the first stage then comprises the complete pseudo
labels for the image $I$, which are utilized in the second stage as ground-
truth labels for training the semantic segmentation network. The following
details the operation of each component.

###  3.2 Inference of Complete Pseudo Labels on a Graph

We begin by defining the graph structure. Let
$\mathcal{G}=(\mathcal{V},\mathcal{E})$ be a graph, where
$\mathcal{V}=\\{v_{1},...,v_{N}\\}$ is a collection of nodes and
$\mathcal{E}=\\{A_{i,j}|v_{i},v_{j}\in\mathcal{V}\\}$ specify edges connecting
nodes $v_{i}$ and $v_{j}$ with weights $A_{i,j}$. In our task, a node
$v_{i}\in\mathbb{R}^{D}$ in $\mathcal{V}$ refers collectively to the co-
located feature samples at $(x_{i},y_{i})$ in $D$ feature maps, each being of
size $H/S\times W/S$ with the total number of nodes $N=H/S\times W/S$. The
choice of node features is detailed in Section 4.1. The pseudo label of
$v_{i}$ is denoted as $p_{i}=P(x_{i},y_{i})$. The edge weight $A_{i,j}$
measures the affinity between $v_{i}$ and $v_{j}$. Because GCN is amenable to
a wide choice of affinity measures, we test two different measures [3, 4] in
our experiments (Section 4.1).

To generate complete pseudo labels $\hat{P}$, we perform feature propagation
and inference on the graph $\mathcal{G}$ through a 2-layer GCN. The feed-
forward inference proceeds as follows:

$Q=\sigma_{s}(\tilde{A}(\sigma_{r}(\tilde{A}VW_{1})W_{2}),$ (1)

where $V=[v_{1},v_{2},...,v_{N}]^{T}\in\mathbb{R}^{N\times D}$ is a matrix of
node feature vectors, each of which is $D$-dimensional;
$W_{1}\in\mathbb{R}^{D\times 16},W_{2}\in\mathbb{R}^{16\times(|C|+1)}$ are the
two learnable network parameters; $\sigma_{r}(\cdot),\sigma_{s}(\cdot)$
correspond to the ReLU and the softmax activation functions, respectively; and
$\tilde{A}=A+I_{N}$ is the sum of the affinity matrix $A\in\mathbb{R}^{N\times
N}$ and an identity matrix $I_{N}$ of size $N$. Note that the number of
classes which include background class is denoted as $|C|+1$. In the resulting
matrix $Q=[q_{1},q_{2},...,q_{N}]^{T}\in\mathbb{R}^{N\times(|C|+1)}$, each row
$q_{i}^{T},i=1,2,\ldots,N$ signals the probability distribution of semantic
classes at pixel $(x_{i},y_{i})$ in the feature domain. These probability
distributions are then interpolated spatial-wise (using bilinear
interpolation) to arrive at a full-resolution semantic prediction map,
followed by applying the dCRF [13] in a channel-wise manner and taking the
maximum across channels at every pixel for complete pseudo labels $\hat{P}$.

###  3.3 Training a GCN for Feature Propagation

The training of a GCN for every image is formulated as a semi-supervised
learning problem and incorporates four loss functions, namely the (1)
foreground loss $\ell_{fg}$, (2) background loss $\ell_{bg}$, (3) entropy loss
$\ell_{ent}$, and (4) Laplacian loss $\ell_{lp}$:

$\ell=\ell_{fg}+\ell_{bg}+\beta_{1}\ell_{ent}+\beta_{2}\ell_{lp},$ (2)

where $\beta_{1}$ and $\beta_{2}$ are hyper-parameters. The first two are
evaluated as the sum of cross entropies over the foreground and background
pixels in the feature domain, respectively, where the foreground pixels have
their partial pseudo labels $P(x,y)\in C$ while the background pixels have
$P(x,y)=c^{bg}$. The rationale behind the separation of the cross entropies
into the foreground and background groups is to address the imbalance between
these two classes of pixels. In symbols, we have

$\ell_{fg}=-\frac{1}{|\mathcal{V}_{fg}|}\sum_{i\in\mathcal{V}_{fg}}{\log{(q_{i})_{p_{i}}}},$
(3)
$\ell_{bg}=-\frac{1}{|\mathcal{V}_{bg}|}\sum_{i\in\mathcal{V}_{bg}}{\log{(q_{i})_{p_{i}}}},$
(4)

where $q_{i}$ from the $Q$ in Eq. (1) denotes the predicted class distribution
at pixel $i$; $(q_{i})_{p_{i}}$ refers specifically to the predicted
probability of the class corresponding to the partial pseudo label $p_{i}$;
$\mathcal{V}_{fg}$ and $\mathcal{V}_{bg}$ are the foreground and background
pixels according to the partial pseudo labels $P$.

For those pixels in the feature domain with their pseudo labels $P(x,y)$
marked as ignored, i.e., unlabeled pixels, we impose the following entropy
loss, requiring that the uncertainty about their class predictions should be
minimized. In other words, it encourages the class predictions $q_{i}$ at
those unlabeled pixels to approximate one-hot vectors.

$\ell_{ent}=-\frac{1}{|\mathcal{V}_{ig}|}\sum_{i\in\mathcal{V}_{ig}}\sum_{c\in\bar{C}}{(q_{i})_{c}\log{(q_{i})_{c}}},$
(5)

where $\bar{C}=C\cup\\{c^{bg}\\}$ and $\mathcal{V}_{ig}$ refers to unlabeled
pixels. In addition, motivated by the observation that neighbouring pixels
with similar color values usually share the same semantic class, we introduce
a Laplacian loss to ensure the consistency of the class predictions with the
image contents. This prior knowledge is incorporated into the training of the
GCN in the form of the Laplacian loss:

$\ell_{lp}=\frac{1}{2|\mathcal{V}|}\sum_{i\in\mathcal{V}}\sum_{j\in\mathcal{V}}{\Phi_{i,j}\|q_{i}-q_{j}\|_{2}^{2}},$
(6)

which aims to minimize the discrepancy (measured in $l_{2}$ norm) between the
class prediction of pixel $i$ and those of its surrounding pixels $j$ in a
neighborhood $\mathcal{N}_{i}$ according to the affinity weight $\Phi_{i,j}$
that reflects the similarity between pixel $i$ and pixel $j$ in terms of their
color values and locations as given by

$\Phi_{i,j}=\begin{cases}\exp(-\dfrac{||\bar{I}_{i}-\bar{I}_{j}||^{2}}{2\sigma_{1}^{2}}-\dfrac{||f_{i}-f_{j}||^{2}}{2\sigma_{2}^{2}})&\mbox{if
}j\in\mathcal{N}_{i}\\\ 0&\mbox{otherwise}\end{cases},$ (7)

where $\bar{I}\in\mathbb{R}^{3\times H/S\times W/S}$ is the bilinearly
downscaled version of the input image $I$; $\bar{I}_{i}$ and
$f_{i}=(x_{i},y_{i})$ refer to the color value at pixel $(x_{i},y_{i})$ and
its $x,y$ coordinates, respectively; $\sigma_{1}=\sqrt{3}$, $\sigma_{2}=10$
are hyper-parameters; and $\mathcal{N}_{i}$ is a $5\times 5$ window centered
at pixel $(x_{i},y_{i})$. Note that $\Phi$, which relies on low-level color
and spatial information to regularize the GCN output, is to be distinguished
from the affinity matrix $A$ in Eq. (1), which uses high-level semantic
information [3, 4] to specify the graph structure of GCN for feature
propagation as will be detailed next.

##  4 Experimental Results

###  4.1 Setup

Datasets and Metrics: Following most of the prior works [3, 4, 5, 6, 7, 10,
11, 12] we evaluate our method on the PASCAL VOC 2012 semantic segmentation
benchmark [14]. It includes images with pixel-wise class labels, of which
1,464, 1,449, and 1,456 are used for training, validation, and test,
respectively. Following the common training strategy for semantic
segmentation, we adopt augmented training images [15], which are composed of
10,582 images annotated by pixel-wise class labels. However, we use only
image-level class labels for weak supervision during training. To evaluate the
semantic segmentation accuracy, we resort to Mean Intersection over Union
(mIoU) as the other baselines.

Implementation and Training: There are two variants, termed WSGCN-I and
WSGCN-P, of our model that use affinity matrices $A$, node features
$\mathcal{V}$ and CAMs from IRNet [4] and PSA [3], respectively. The WSGCN-I
constructs the affinity matrix $A$ using the boundary detection network [4]
and takes as node features $\mathcal{V}$ the features that sit in the last
layer of the boundary detection network and before its $1\times 1$
convolutional layer. In contrast, the WSGCN-P follows AffinityNet [3] in
specifying the affinity matrix $A$ and uses the semantic features for affinity
evaluation as node features $\mathcal{V}$. Both variants apply the same CAMs
as their counterparts in IRNet [4] and PSA [3]. For every training image, we
train both variants for $250$ gradient update steps with a dropout rate of
$0.3$. The $\beta_{1},\beta_{2}$ in Eq. (2) are set to $10$ and $10^{-2}$,
respectively. The learning rate of the optimizer Adaptive Moment Estimation
(Adam) is set to $0.01$, with a weight decay of $5\times 10^{-4}$. For a fair
comparison, we follow [6, 7, 10] to adopt DeepLabv2 [16] as our semantic
segmentation network in the second stage. The backbone ResNet-101 is pre-
trained on ImageNet.

Baselines: We compare our approach with several state-of-the-art weakly-
supervised image semantic segmentation methods [3, 4, 5, 6, 7, 10, 11, 12] on
training, validation, and test sets. The comparison with PSA [3] and IRNet [4]
singles out the benefits of our feature propagation framework over label
propagation since our GCN adopts the same affinity matrix as theirs for
propagating features rather than activation scores of CAMs.

Table 1: Quality comparison of complete pseudo labels on PASCAL VOC 2012 training set. Method | mIoU (%)
---|---
PSA [3] | 59.7
SC-CAM [6] | 63.4
SEAM [5] | 63.6
IRNet [4] | 66.5
SingleStage [7] | 66.9
WSGCN-P | 64.0
WSGCN-I | 68.0

###  4.2 Quality of Complete Pseudo Labels

Table 1 presents a quality comparison of complete pseudo labels between our
WSGCN-I/P and several state-of-the-art methods [3, 4, 5, 6, 7, 11, 12] in
terms of mIoU on the PASCAL VOC 2012 training set. Most of these baselines
propagate activation scores of CAMs with random walk (i.e., label propagation)
to infer complete pseudo labels.

We see that our WSGCN-I achieves the highest mIoU (68.0%) among all the
competing methods. It shows a 1.5% mIoU improvement over IRNet [4]. Likewise,
WSGCN-P has a 4.3% mIoU gain as compared to PSA [3]. These results confirm the
superiority of our feature propagation framework over the label propagation
adopted by these baselines.

Further insights can be gained by noting that in generating complete pseudo
labels for every training image, our schemes rely on training a specific GCN
by back-propagation. On the contrary, label propagation is a feed-forward
process. During back-propagation, our schemes involve not only high-level node
features $\mathcal{V}$ in constructing an affinity matrix $A$ for feature
propagation but also low-level color/spatial information (cp. $\Phi$ in
$\ell_{lp}$) in regularizing label predictions, whereas the label propagation
used by the baselines depends largely on high-level node features
$\mathcal{V}$ to compute a Markov transition matrix. A side experiment shows
that substituting the affinity matrix $A$ in Eq. (1) for $\Phi$ in $\ell_{lp}$
causes the mIoU of complete pseudo labels to decrease from $68\%$ to $66.3\%$.
This is in agreement with the general observation that high-level semantic
features usually contain less information about spatial details. Note also
that the performance gap between WSGCN-I and WSGCN-P comes from the use of
different affinity matrices $A$ and node features $\mathcal{V}$, emphasizing
the influence of these design choices on the quality of complete pseudo
labels.

###  4.3 Ablation Study of Loss Functions

Table 2 further analyzes how the single use of various loss functions and
their combinations along with dCRF contribute to the quality of complete
pseudo labels. By default, the foreground and background losses are enabled.
They alone (WSGCN-I without any check mark) offer $62.0\%$ mIoU. The entropy
loss improves the mIoU further by $1.5\%$. The Laplacian loss attains even
higher gain ($2.4\%$) due to the incorporation of low-level color/spatial
regularization. When combined, they show a synergy effect of $4.7\%$. Recall
that the entropy loss encourages the GCN to produce a one-hot-like semantic
prediction while the Laplacian loss requires the predictions to be close to
each other for adjacent pixels with similar colors. It is interesting to note
that dCRF (as a post-processing step) can further improve on the entropy loss,
the Laplacian loss as well as their combination. Although dCRF uses similar
low-level information to the Laplacian loss, it functions as a separate,
refinement step rather than a substitute for the Laplacian loss. Unlike dCRF,
the Laplacian loss is included in the training loop of GCN.

Fig. 2 visualizes how these loss functions improve incrementally the quality
of complete pseudo labels. We see that based on the semantic segmentation
resulting from the foreground and background losses, the entropy loss tends to
grow the foreground regions while the Laplacian loss can help alleviate
semantic segmentation errors at object boundaries.

Table 2: Effect of various loss functions on the quality of complete pseudo labels evaluated on PASCAL VOC 2012 training set. Method | Entropy | Laplacian | dCRF | mIoU (%)
---|---|---|---|---
loss | loss | (post-processing)
WSGCN-I |  |  |  | 62.0
WSGCN-I | $\surd$ |  |  | 63.5
WSGCN-I |  | $\surd$ |  | 64.4
WSGCN-I | $\surd$ | $\surd$ |  | 66.7
WSGCN-I | $\surd$ |  | $\surd$ | 66.0
WSGCN-I |  | $\surd$ | $\surd$ | 66.1
WSGCN-I | $\surd$ | $\surd$ | $\surd$ | 68.0
Partial pseudo label |  |  |  |
---|---|---|---|---
$\ell_{fg}+\ell_{bg}$ |  |  |  |
$\ell_{fg/\/bg}+\ell_{ent}$ |  |  |  |
WSGCN-I |  |  |  |
IRNet |  |  |  |
GT |  |  |  |
Fig. 2: Sample results of pseudo labels. From top to bottom: partial pseudo labels where white regions indicate unlabeled pixels, complete pseudo labels by $\ell_{fg}+\ell_{bg}$, complete pseudo labels by $\ell_{fg}+\ell_{bg}+\ell_{ent}$, complete pseudo labels by all four losses (WSGCN-I), complete pseudo labels by IRNet, and the ground-truths. All complete pseudo labels are refined with dCRF. Table 3: Comparison of semantic segmentation accuracy evaluated on PASCAL VOC 2012 dataset.  Method | Publication | Backbone | mIoU (%)
---|---|---|---
$val$ | $test$
PSA [3] | CVPR18 | ResNet-38 | 61.7 | 63.7
IRNet [4] | CVPR19 | ResNet-50 | 63.5 | 64.8
SEAM [5] | CVPR20 | ResNet-38 | 64.5 | 65.7
SC-CAM$\dagger$ [6] | CVPR20 | ResNet-101 | 66.1 | 65.9
RRM$\dagger$ [10] | AAAI20 | ResNet-101 | 66.3 | 66.5
SingleStage$\dagger\dagger$ [7] | CVPR20 | ResNet-101 | 65.7 | 66.6
MCIS* [11] | ECCV20 | ResNet-101 | 66.2 | 66.9
GroupWSSS* [12] | AAAI21 | ResNet-101 | 68.2 | 68.5
WSGCN-I (ours) | – | ResNet-101 | 66.7 | 68.8
WSGCN-I$\dagger$ (ours) | – | ResNet-101 | 68.7 | 69.3
*: Using saliency maps as extra supervision signals.
$\dagger$: Pre-training the backbone in DeepLabv2 on MS-COCO.
$\dagger\dagger$: Using DeepLabv3+ as the semantic segmentation model.
Table 4: Comparison of PSA, IRNet, and our method with DRN-D-105 backbone.  Method | Backbone | mIoU (%)
---|---|---
$val$ | $test$
PSA [3] | DRN-D-105 | 60.9 | 61.9
WSGCN-P (ours) | DRN-D-105 | 63.1 | 64.4
IRNet [4] | DRN-D-105 | 66.0 | 66.5
WSGCN-I (ours) | DRN-D-105 | 66.9 | 67.5
Table 5: Comparison of WSGCN-I and IRNet with the ResNet-101 backbone on the PASCAL VOC 2012 test set. Method | bkg | aero | bike | bird | boat | bot. | bus | car | cat | chair | cow | tab. | dog | horse | mbk | per. | plant | sheep | sofa | train | tv | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
IRNet [4] | 90.2 | 75.8 | 32.5 | 76.5 | 49.6 | 65.9 | 82.6 | 76.2 | 82.7 | 31.8 | 77.5 | 41.8 | 79.4 | 75.0 | 81.2 | 75.2 | 47.9 | 81.1 | 51.5 | 62.5 | 59.1 | 66.5
WSGCN-I | 91.0 | 79.8 | 33.8 | 78.2 | 50.9 | 65.7 | 86.8 | 79.3 | 86.0 | 27.4 | 75.2 | 48.9 | 83.4 | 76.2 | 82.7 | 74.4 | 64.3 | 86.0 | 51.0 | 64.2 | 58.6 | 68.8

###  4.4 Semantic Segmentation Performance

Using complete pseudo labels as supervision, we train a semantic segmentation
network in the second stage. Table 3 compares the semantic segmentation
accuracy of our 2-stage training framework with GCN-based feature propagation
to several state-of-the-art methods. We see that our WSGCN-I outperforms all
the baselines which do not use saliency maps on both validation and test
datasets, with the ResNet-101 backbone pre-trained on ImageNet. It gains 0.5%
(respectively, 2%) more mIoU on the test (respectively, validation) dataset,
when pre-training the backbone on MS-COCO dataset as in [6, 10].

Table 4 further compares WSGCN-P and WSGCN-I with their counterparts PSA [3]
and IRNet [4], respectively. Note that the main difference between WSGCN-P and
WSGCN-I is the choice of the affinity matrix (Section 4.1). For a fair
comparison, we choose DRN-D-105 [17], which is pre-trained on ImageNet and is
publicly accessible for reproducibility, as the backbone in DeepLabv2. It is
seen that these variants show $0.9\%$ to $2.5\%$ mIoU improvements, confirming
the superiority of our feature propagation framework to label propagation. The
results also demonstrate that our scheme can work with different affinity
matrices.

Table 5 additionally compares our WSGCN-I and IRNet [4] with the ResNet-101
backbone. As shown, WSGCN-I achieves higher IoU in most categories than IRNet
[4] and has a $2.3\%$ mIoU improvement.

##  5 Conclusion

This paper presents a GCN-based feature propagation framework for weakly-
supervised image semantic segmentation. Unlike feed-forward label propagation,
our approach optimizes the generation of complete pseudo labels by learning
off-line a separate GCN for every training image. The learning process
involves feature propagation based on high-level semantic affinity between
pixels and label prediction regularized by low-level spatial coherence. On our
computation platform with 4 NVIDIA Tesla V100 GPUs and 1 Intel Xeon 3GHz CPU,
the runtime needed to infer complete pseudo labels for one sample image is
about 2.45s, as compared to 1.43s for label propagation with IRNet [4].
Experimental results on PASCAL VOC 2012 benchmark validate the superiority of
our framework to label propagation.

## References

  * [1] Yunchao Wei, Huaxin Xiao, Humphrey Shi, Zequn Jie, Jiashi Feng, and Thomas S. Huang,  “Revisiting dilated convolution: A simple approach for weakly- and semi-supervised semantic segmentation,”  in CVPR, 2018.
  * [2] Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang,  “Weakly-supervised semantic segmentation network with deep seeded region growing,”  in CVPR, 2018.
  * [3] Jiwoon Ahn and Suha Kwak,  “Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation,”  in CVPR, 2018.
  * [4] Jiwoon Ahn, Sunghyun Cho, and Suha Kwak,  “Weakly supervised learning of instance segmentation with inter-pixel relations,”  in CVPR, 2019.
  * [5] Yude Wang, Junfeng Zhang, Meina Kan, Shiguang Shan, and Xilin Chen,  “Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation,”  in CVPR, 2020.
  * [6] Yu-Ting Chang, Qiaosong Wang, Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, and Ming-Hsuan Yang,  “Weakly-supervised semantic segmentation via sub-category exploration,”  in CVPR, 2020.
  * [7] Nikita Araslanov and Stefan Roth,  “Single-stage semantic segmentation from image labels,”  in CVPR, 2020.
  * [8] Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba,  “Learning deep features for discriminative localization,”  in CVPR, 2016.
  * [9] Thomas Kipf and Max Welling,  “Semi-supervised classification with graph convolutional networks,”  in ICLR, 2017.
  * [10] Bingfeng Zhang, Jimin Xiao, Yunchao Wei, Mingjie Sun, and Kaizhu Huang,  “Reliability does matter: An end-to-end weakly supervised semantic segmentation approach,”  in AAAI, 2020.
  * [11] Guolei Sun, Wenguan Wang, Jifeng Dai, and Luc Van Gool,  “Mining cross-image semantics for weakly supervised semantic segmentation,”  in ECCV, 2020.
  * [12] Xueyi Li, Tianfei Zhou, Jianwu Li, Yi Zhou, and Zhaoxiang Zhang,  “Group-wise semantic mining for weakly supervised semantic segmentation,”  in AAAI, 2021.
  * [13] Philipp Krähenbühl and Vladlen Koltun,  “Efficient inference in fully connected crfs with gaussian edge potentials,”  in NIPS, 2011.
  * [14] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman,  “The pascal visual object classes (VOC) challenge,”  IJCV, vol. 88, pp. 303–338, 2009.
  * [15] B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, and J. Malik,  “Semantic contours from inverse detectors,”  in ICCV, 2011.
  * [16] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille,  “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,”  IEEE TPAMI, vol. 40, pp. 834–848, 2018.
  * [17] Fisher Yu, Vladlen Koltun, and Thomas Funkhouser,  “Dilated residual networks,”  in CVPR, 2017.