# Extracting Semantic Knowledge from GANs with Unsupervised Learning

Jianjin Xu, Zhaoxiang Zhang, Xiaolin Hu  J. Xu is with the School of
Mathematics and Computer Science, Panzhihua University and the Department of
Computer Science and Technology, Tsinghua University.
E-mail<EMAIL_ADDRESS>Z. Zhang is with with the Insitute of Automation,
Chinese Academy of Sciences and the Centre for Artificial Intelligence and
Robotics, Hong Kong Innovation and Science Insitute, Chinese Academy of
Sciences.
E-mail<EMAIL_ADDRESS>X. Hu is with the Department of Computer
Science and Technology, Institute for Artificial Intelligence, State Key
Laboratory of Intelligent Technology and Systems, THU-Bosch JCML Center, THBI,
IDG/McGovern Institute for Brain Research, BNRist, Tsinghua University,
Beijing, China. He is also with the Chinese Institute for Brain Research
(CIBR), Beijing, China.
E-mail<EMAIL_ADDRESS>work was supported in part by the National
Natural Science Foundation of China (Nos. 62061136001, 61836014, and U19B2034)
and THU-Bosch JCML center. Corresponding author: Xiaolin Hu.

###### Abstract

Recently, unsupervised learning has made impressive progress on various tasks.
Despite the dominance of discriminative models, increasing attention is drawn
to representations learned by generative models and in particular, Generative
Adversarial Networks (GANs). Previous works on the interpretation of GANs
reveal that GANs encode semantics in feature maps in a linearly separable
form. In this work, we further find that GAN’s features can be well clustered
with the linear separability assumption. We propose a novel clustering
algorithm, named KLiSH, which leverages the linear separability to cluster
GAN’s features. KLiSH succeeds in extracting fine-grained semantics of GANs
trained on datasets of various objects, _e.g_., car, portrait, animals, and so
on. With KLiSH, we can sample images from GANs along with their segmentation
masks and synthesize paired image-segmentation datasets. Using the synthesized
datasets, we enable two downstream applications. First, we train semantic
segmentation networks on these datasets and test them on real images,
realizing unsupervised semantic segmentation. Second, we train image-to-image
translation networks on the synthesized datasets, enabling semantic-
conditional image synthesis without human annotations.

###### Index Terms:

GAN, Unsupervised Learning, Semantic Segmentation, Conditional Image Synthesis

##  1 Introduction

Representation is at the core of modern deep learning practice. Bengio _et
al_. [1] propose that a good representation should have (1) _explanatory
factors organized in hierarchy_ , and (2) _natural clustering of categories_.
Chan _et al_. [2] claim that the objective of a deep network is to learn a
linearly discriminative representation of the data. Researchers have found
that the representation learned by GANs [3] not only explains the variation of
images [4, 5, 6] but also is linearly discriminative [7], indicating that GANs
learn a good representation.

Given a dataset of images, a GAN learns to model the distribution of data and
synthesizes images unseen in the dataset. The image synthesis process of GAN
is typically a mapping from noise vectors to a series of intermediate feature
maps and then to RGB images. From the perspective of causality, GAN’s
representations contain all the information in the image. In other words, the
variation factors (_e.g_., the attributes of objects) present in the image
must also exist in GAN’s representation. Previous works have identified two
types of variations captured by GANs. First, the image-level attributes
(_e.g_., the gender of face images) are separable by linear hyperplanes in
GAN’s latent space [4, 8, 5, 9, 10]. Second, the object-level semantics
(_e.g_., the presence and location of a window in bedroom images) are encoded
in GAN’s feature maps [6] in a linearly separable form [7]. We term the rich
semantic information in GAN’s features as the semantic knowledge learned by
the GANs.

Figure 1: Semantic segmentation obtained by using the average feature of each
class compared to using a trained linear classifier. The prediction from a
pretrained DeepLabV3 [11] is regarded as the groundtruth. The linear
classifier is supervised by a DeepLabV3 network pretrained on CelebAMask-HQ
dataset [12] according to Xu _et al_. [7].

Existing works [13, 14, 15] often train few-shot learning classifiers to
reveal GAN’s semantic knowledge. The classifiers predict each spatial location
of GAN’s feature maps. Training the classifiers needs only a few labeled
examples. The trained classifier is subsequently used for image editing [13,
14] and semantic segmentation [15]. However, the labeling cost for the few-
shot learning method is still high. As reported in [13, 14], the number of
annotations ranges from 16 to 40 and labeling each image takes around 10
minutes even for a skillful user. Therefore, if one can design an unsupervised
segmentation algorithm based on GAN’s feature maps for image editing, it would
enable users to upload their own images, train their own GANs, and edit those
images without much labor cost. This is the aim of this study. Specifically,
we study how to reveal GAN’s semantic knowledge with unsupervised learning
methods.

Existing unsupervised learning methods mostly apply conventional clustering
algorithms on GAN’s features, _e.g_., K-means [16, 17] and Agglomerative
Hierarchical Clustering (AHC) [18]. They use clustering results as an
intermediate step for editing the attribute [17] or swapping the style [16] in
a localized region. However, the obtained clusters are not satisfactory.
Please refer to their results [16, 17, 18] and also our reproduction in Fig.
5. For example, for face images, we observed the oversegmentation of the hair
class, and the missing of tiny classes like brow, upper lip, and lower lip
(see Fig. 5, the last column).

Figure 2:  An illustration of clustering on linearly separable data. Top-left, the groundtruth cluster assignments, and the decision boundaries of the linear SVM trained on the groundtruth. Other panels, results of different clustering algorithms. Decision boundaries are visualized as solid lines, except for AHC which has no such boundary. “x” denotes the centroids of K-means. GAN | StyleGAN2 | StyleGAN | PGGAN
---|---|---|---
Dataset | FFHQ | CelebAHQ | CelebAHQ
Linear Classifier [7] | 82.7 | 73.8 | 69.3
Centroids of Features | 49.6 | 46.6 | 41.8
TABLE I: The semantics segmentation performance (measured in mIoU) of linear
classifier or centroids of features. Please see Sec. 4.2 for details.

The reason might be that K-means and AHC require data to be centered around
their class centroids, which is not well satisfied in GAN’s features. To
illustrate this, we first took a pretrained GAN and segmented the generated
images with a pretrained DeepLabV3 [11] model. The features from the GAN were
averaged into a centroid for each semantic class. Then, we segmented another
set of generated images by assigning the closest centroid to GAN’s features on
each spatial location. It was observed that the segmentation results were
rather coarse (Fig. 1).

Interestingly, a previous work [7] found that GAN’s features of a semantic
class were linearly separable from other classes. This observation is referred
to as the GAN’s linear separability in semantics. We also found that the
linear classifier produced better segmentation than using the centroid method
(Tab. I). Therefore, we propose to cluster GAN’s features by leveraging their
linear separability property, instead of the conventional centering
assumption.

We note that linearly separable data does not necessarily satisfy the
centering assumption required by conventional clustering algorithms. To
illustrate this, we generated three sets of points in a 2D plane, which were
linearly separable from each other (Fig. 2, top-left). In this synthetic
dataset, each cluster had a complex structure and its centroid had limited
information about the structure. It was observed that both AHC and K-means
assigned a part of the blue cluster to the red cluster.

We propose K-means with Linear Separability Heuristic (KLiSH) to find
semantically meaningful clusters in GAN’s features. In brief, KLiSH starts
with a large number of initial K-means clusters and iteratively merges the
clusters according to their degree of linear separability. KLiSH successfully
clustered the three sets of points on the toy dataset described above (Fig. 2,
bottom-right). We compared KLiSH to several clustering algorithms, AHC,
K-means, and K-means based Approximate SPectral clustering (KASP) [19] on
several GANs and datasets. Results showed that KLiSH outperformed all
baselines in most cases.

One can use KLiSH to obtain fine-grained semantic segmentation for the
generated images. The process is straightforward. First, sample images from
GANs and apply the classifier returned by KLiSH on GAN’s feature maps. Then,
train downstream models on the resulting image-segmentation dataset. We
demonstrate two applications in this paper: Unsupervised Fine-Grained
Segmentation (UFGS) and Unsupervised Semantic-Conditional Synthesis (USCS).
UFGS seeks to reduce the labeling cost of fine-grained semantic annotation on
a new dataset. USCS aims to offer semantic-controllable image synthesis and
editing on datasets without semantic annotation. Both UFGS and USCS can be
realized easily with the semantic knowledge extracted from pretrained GANs.

Figure 3: The pipeline of re-purposing pretrained GANs for downstream tasks,
UFGS and USCS. First, we propose the K-means with Linear Separability
Heuristic (KLiSH) to cluster GAN’s features. Then, we generate datasets with
synthesized images and segmentation masks together from GANs. Finally, we
train different models on the synthesized datasets and perform semantic
segmentation on _real_ images and semantic-conditional image synthesis. Both
applications require no human annotations at all.

The pipeline of UFGS and USCS is illustrated in Fig. 3. Given a dataset of
images, we first train a GAN on it and use KLiSH to parse the semantic
clusters learned by GANs. Then, we generate a paired image-segmentation
dataset by sampling images from GANs along with the segmentation masks.
Finally, we train corresponding downstream models on the generated dataset.

In summary, our contributions are as follows. First, we propose KLiSH and
improve the quality of clustering on GAN’s features. Second, we provide
evidence for GAN’s linear separability in semantics. Third, we propose two
novel applications, UFGS and USCS, marking a successful attempt at fine-
grained semantic segmentation and semantic-conditional synthesis in an
unsupervised learning setting.

##  2 Related work

#### Generative Adversarial Networks

Originally put forward by Goodfellow _et al_. [3], GANs go through many
improvements in training [20, 21], architecture [22, 23, 24, 25] and objective
[26, 27, 28, 29]. Nowadays, GANs can map latent vectors to diverse and photo-
realistic images on various datasets. Typical data domains are face [30, 31,
32], car [33, 34, 20], animal [35, 36], ImageNet [37, 38], _etc_.. Among
various GANs, StyleGAN2 [34] is one of the state-of-the-art models. Another
genre of GANs translates images to images [39, 40, 41]. When the input image
is a semantic mask, the task is termed Semantic-Conditional Synthesis (SCS).
Typical models capable of SCS include pix2pix [42], pix2pixHD [43], GauGAN
[44] and so on [45, 12, 46, 47]. Most of them heavily rely on large-scale
semantically-annotated datasets. Sun _et al_. [48] propose to synthesize
objects in an image from bounding boxes, which lowers the label requirement,
but still needs a large-scale dataset. In contrast, the USCS proposed in this
work is fully unsupervised – it is realized by making use of the
representations learned by pretrained GANs.

#### GAN interpretation

Increasing attention is placed on studying GAN’s representation. Most existing
works focus on studying GAN’s latent vectors and feature maps. For the first
aspect, prior works [4, 5] find that there are linear boundaries in latent
space that separate the positive and negative attributes of the sampled
images. Some other works focus on the unsupervised discovery of those
boundaries [8, 10, 9]. For the second aspect, GAN Dissection [6] identifies
convolution units that have causal relationships with semantics in the
generated images. Xu and Zheng [7] propose and validate that GANs encode high-
resolution semantics linearly in feature maps. Other works study the
parameters of GANs [49, 50]. This work proposes an unsupervised approach to
extract learned semantic classes from GANs, providing strong evidence that the
GAN learns a naturally clustered and linearly separable semantic
representation.

#### Repurposing GAN for semantic segmentation

Recently, increasing attention is directed towards semantic segmentation using
GAN’s representation. Existing works either fall into few-shot learning or
unsupervised learning. The few-shot learning methods typically apply
classifiers on a pretrained GAN’s feature maps to get semantic segmentation
for the generated image [7, 51, 15, 14]. These methods can segment the images
well with a limited number of image annotations (ranging from 1 to 40
depending on the complexity of the target data).

For unsupervised learning, the majority of methods [52, 53, 54, 55, 56, 57]
focus on decoupling the foreground and background in GANs without external
labeling. Though the segmentation is accurate, the practical usage is limited
as only two clusters are segmented. Methods on obtaining more clusters mostly
involve applying K-means++ [16] or AHC [18, 17] directly on GAN’s features. He
_et al_. [58] modify K-means by re-weighting channels. Different from existing
methods, we propose a new clustering algorithm, KLiSH, by leveraging a strong
characteristic of GAN’s features – linear separability. We conducted
quantitative evaluations and showed the superiority of KLiSH to previous
clustering algorithms on GAN’s features.

Aside from GAN, other works [59, 60, 61] also study unsupervised semantic
segmentation on large-scale datasets, but mainly focus on object-level
semantics instead of part-level semantics and their results are not directly
comparable to GAN-based methods.

##  3 Clustering Generator’s Features

###  3.1 Linear Separability of GAN’s Features

#### Classical linear separability

KLiSH is based on GAN’s linear semantics [7], thus we first introduce this
characteristic to make our paper self-contained.

When the generator synthesizes an image, it also produces a set of feature
maps. A $1\times 1$ convolutional layer is applied to the upsampled and
concatenated feature maps, resulting in a score map of semantic classes for
the generated image. As the extraction process is fully linear, the method is
named Linear Semantic Extractor (LSE). The LSE is compared to two nonlinear
semantic extractors (NSEs) with more complicated architectures. The LSE and
NSEs are trained under similar settings and evaluated by the accuracy of the
semantic masks they extracted compared to the pretrained segmentation network.
Results show that the relative performance gaps between LSE and NSEs are
within $3.5\%$ across various datasets and models. Therefore, it is shown that
linear classifiers suffice to separate GAN’s representations of different
semantics. For more detailed settings and results, we refer readers to the
original paper [7].

The phenomenon that a linear classifier can classify a given dataset is
usually described as linearly separable. Given a dataset
$\mathcal{A}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{N}$ where
$\mathbf{x}_{i}\in\mathbb{R}^{D}$ and $y_{i}\in\\{1,\ldots,M\\}$, the
classical linear separability states that there exists a set of hyperplanes
$\mathbf{W}\in\mathbb{R}^{M\times D}$ and $\mathbf{b}\in\mathbb{R}^{D}$ such
that

$\forall(\mathbf{x},y)\in\mathcal{A},k\neq
y,\mathbf{W}_{y}\mathbf{x}+\mathbf{b}_{y}>\mathbf{W}_{k}\mathbf{x}+\mathbf{b}_{k},$
(1)

where $\mathbf{W}_{k}\in\mathbb{R}^{1\times D}$ is a row vector, and
$\mathbf{W}_{y}$ denotes the classifier weight for the ground-truth class of a
data point $\mathbf{x}$.

#### Binary linear separability

Instead of using classical linear separability, we find that binary linear
separability is more useful for building our clustering algorithm, which is
defined below.

###### Definition 1

Given a dataset $\mathcal{A}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{N}$, we say
$\mathcal{A}$ satisfies the binary linear separability if there exists
$\mathbf{W}\in\mathbb{R}^{M\times D}$ and $\mathbf{b}\in\mathbb{R}^{D}$ such
that $\forall(\mathbf{x}_{i},y_{i})\in\mathcal{A}$, $k\in\\{1,\dots,M\\}$,

$\mathcal{I}(y_{i}=k)\cdot(\mathbf{W}_{k}\mathbf{x}_{i}+\mathbf{b}_{k})>1,$
(2)

where $\mathcal{I}(y_{i}=k)$ returns 1 if $y_{i}=k$ and -1 otherwise.

In other words, if for each class in the dataset, there is a two-class SVM
separating itself from others, then the dataset satisfies the binary linearly
separability. Unless otherwise specified, linear separability refers to
Definition 1 throughout the rest of the paper.

Clearly, Eq. 2 is harder to satisfy than Eq. 1. Clusters separable with a
classical linear classifier might not be separable with binary SVMs. Do
features in GAN satisfy the binary linear separability? We empirically found
that the answer is yes. The experiment is presented in Sec. 4.2.

###  3.2 Clustering with Linear Separability Heuristic

Now, we describe the algorithm for clustering GAN’s features by linear
separability.

#### Initialize from K-means

We choose K-means for three reasons. First, K-means is easily scalable to
large datasets with time complexity of $O(N)$. Second, the decision boundaries
between clusters are linear and thus are suitable to implement linear
separability. Third, K-means with a large number of clusters provides a good
initialization for our algorithm. More specifically, most clusters found by
K-means do not span across multiple semantic classes, see the K-means@100
column in Fig. 5. This simplifies our problem as we only need to merge these
clusters to recover the ground-truth classes.

#### Merge clusters

Ideally, the merged clusters should belong to the same semantic class.
However, we have no access to semantic information as the algorithm is fully
unsupervised. We make a reasonable assumption: the more linearly separable a
cluster is, the more likely it is a true semantic class. Under this
assumption, our goal is to maximize the linear separability of clusters.

How to quantify the degree of linear separability? According to Definition 1,
it is implied that how accurately the SVM classifies a cluster indicates how
well its linear separability is. Therefore, we use the performance of the SVM
trained on the target dataset to indicate its linear separability.

First, a pseudo-labeled dataset $\mathcal{A}$ with cluster assignments is
constructed. Second, binary SVMs are trained on $\mathcal{A}$ using the
L2-regularized L2-loss Support Vector Classification objective [62]:

$\mathcal{L}=\frac{\lambda_{1}}{M|\mathcal{A}|}\sum_{(\mathbf{x},y)\in\mathcal{A}}\bigg{[}(1-\mathbf{W}_{y}\mathbf{x}-\mathbf{b}_{y})_{+}^{2}\\\
+\sum_{k\neq
y}(1+\mathbf{W}_{k}\mathbf{x}+\mathbf{b}_{k})_{+}^{2}\bigg{]}+\frac{1}{2M}\left\lVert\mathbf{W}\right\rVert_{F}^{2},$
(3)

where $(\cdot)_{+}$ denotes $max(0,\cdot)$ and
$\left\lVert\cdot\right\rVert_{F}$ denotes the Frobenius norm. We use L-BFGS
[63] to minimize $\mathcal{L}$ due to the convexity of the objective. The SVM
is considered as converged if its $L_{\infty}$ norm of weight changing is
below $10^{-4}$. Third, as in semantic segmentation, we measure the
performance of SVM by IoU (Intersection-over-Union) defined as

$\displaystyle\text{IoU}_{k}(\mathbf{W},\mathcal{A})=\frac{|\mathcal{S}_{k}\cap\mathcal{Y}_{k}|}{|\mathcal{S}_{k}\cup\mathcal{Y}_{k}|},$
(4)

where
$\mathcal{S}_{k}=\\{\mathbf{x}_{i}|\mathbf{W}_{k}\mathbf{x}_{i}+\mathbf{b}_{k}>0\\}$
is the prediction given by the SVM,
$\mathcal{Y}_{k}=\\{\mathbf{x}_{i}|y_{i}=k\\}$ is the set of positive samples
for class $k$, and $(\mathbf{x}_{i},y_{i})\in\mathcal{A}$. Therefore, we can
use IoU to find the cluster with the lowest linear separability.

Suppose that the identified cluster is $p$. The next step is to find another
cluster $q$ to merge with $p$. As implied by Eq. 4, the cluster $p$ is not
classified accurately by the SVM, _i.e_., the SVM might confuse $p$ with other
clusters. Then a reasonable strategy is to merge $p$ with another cluster that
confuses it the most. To quantify the degree of confusion, we propose the
Effective Cosine Similarity (ECoS) metric as defined below.

###### Definition 2

Consider a dataset $\mathcal{A}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{N}$.
Suppose a set of binary SVMs with weight $\mathbf{W}\in\mathbb{R}^{M\times D}$
and bias $\mathbf{b}\in\mathbb{R}^{D}$ are trained on $\mathcal{A}$ until
convergence. The ECoS between cluster $i$ and cluster $j$ is defined as

$\begin{split}&\text{ECoS}_{i,j}(\mathbf{W},\mathcal{A})=\cos\langle\mathbf{s}_{i}^{\dagger},\mathbf{s}_{j}^{\dagger}\rangle\\\
&\mathbf{s}_{k}^{\dagger}=\text{CLAMP}_{[0,1]}\left(\frac{\mathbf{s}_{k}+1}{2}\right)\\\
&\mathbf{s}_{k}=\left[\mathbf{x}_{1}^{T}\mathbf{W}_{k}+\mathbf{b}_{k},\ldots,\mathbf{x}_{N}^{T}\mathbf{W}_{k}+\mathbf{b}_{k}\right]\end{split}$
(5)

where $\text{CLAMP}_{[a,b]}$ denotes clamping the input within $[a,b]$.

Here, $\mathbf{s}_{k}$ is the cluster scores predicted by the SVM.
$\mathbf{s}_{k}^{\dagger}$ scales and truncates $\mathbf{s}_{k}$ within [0,
1], representing the confidence of SVM predicting positive data points. As
each feature vector corresponds to a pixel, $\mathbf{s}_{k}^{\dagger}$ can be
resized into a cluster confidence map. In other words, ECoS measures the
cosine similarity between two cluster confidence maps.

#### Filter initial clusters

We observed in experiments that a small portion of initial clusters did span
across multiple semantic classes. For example, see K-means@100 of
StyleGAN2-FFHQ in Fig. 5, the boundary around ear (visualized as a yellow
region) contains part of “ear”, “hair”, and “face”. This type of initial
clusters would harm the accuracy of merged clusters. Fortunately, we
empirically found that these clusters were not linearly separable, _i.e_.,
could be filtered by setting a threshold of IoU. The threshold could also be
determined adaptively as described below.

First, train SVMs on the initial clusters and calculate their IoU, resulting
in a vector $\mathbf{m}$. Second, $\mathbf{m}$ is converted from [0, 1] to
$(-\infty,\infty)$ by using the inverse sigmoid function $h^{-1}(\cdot)$.
Third, any clusters below $\mu_{\mathbf{m}}-\sigma_{\mathbf{m}}$ are filtered,
where $\mu$, $\sigma$ are the mean and standard deviation of
$h^{-1}(\mathbf{m})$, respectively. Finally, K-means is run again with the
filtered centroids as initialization.

Input: $\mathcal{D}=\\{\mathbf{x}_{i}\\}_{i=1}^{N}$; $K_{0}$

Output: $\Theta=\\{(\mathbf{W}^{t},\mathbf{b}^{t})\\}_{t=1}^{K_{0}-1}$

$\mathbf{W}^{0},\mathbf{b}^{0},\mathcal{A}^{0}$ = K-means($\mathcal{D},K_{0}$)

// Filter initial centriods

$\mathbf{W}^{\prime},\mathbf{b}^{\prime}$ =
TrainSVM($\mathbf{W}^{0},\mathbf{b}^{0},\mathcal{A}^{0}$)

$\mathbf{m}=\left[h^{-1}(\text{IoU}_{p}(\mathbf{W}^{\prime},\mathbf{b}^{\prime},\mathcal{A}^{0}))\right]_{p=1}^{K_{0}}$

$\mathbf{W}^{0},\mathbf{b}^{0},\mathcal{A}^{0}\leftarrow\text{K-means}(\mathcal{D},\\{\mathbf{W}_{p}^{0}|\mathbf{m}_{p}>\mu_{\mathbf{m}}-\sigma_{\mathbf{m}}\\})$

for  _$t=1,\ldots,K_{0}-1$_ do

$\mathbf{W}^{t},\mathbf{b}^{t}$ =
TrainSVM($\mathbf{W}^{t-1},\mathbf{b}^{t-1},\mathcal{A}^{t-1}$)

$\Theta\leftarrow\Theta\cup\\{(\mathbf{W}^{t},\mathbf{b}^{t})\\}$

$p^{t},\phi^{t}=\min_{p}\text{IoU}_{p}(\mathbf{W}^{t},\mathcal{A}^{t-1})$

$q^{t},\psi^{t}=\max_{q}\text{ECoS}_{p^{t},q}(\mathbf{W}^{t},\mathcal{A}^{t-1})$

// Merge cluster $p^{t}$ into $q^{t}$

$\mathcal{A}^{t}=\\{(\mathbf{x}_{i},q^{t})|y_{i}=p^{t}\\}\cup\\{(\mathbf{x}_{i},y_{i})|y_{i}\neq
p^{t}\\}$

$\mathbf{W}^{t}\leftarrow\text{cat}(\ldots,\mathbf{W}_{p^{t}-1}^{t},\mathbf{W}_{p^{t}+1}^{t},\ldots)$

$\mathbf{b}^{t}\leftarrow\text{cat}(\ldots,\mathbf{b}_{p^{t}-1}^{t},\mathbf{b}_{p^{t}+1}^{t},\ldots)$

Algorithm 1 The KLiSH clustering algorithm.

#### KLiSH algorithm

The full KLiSH algorithm is presented in Algorithm 1, where $\mathcal{D}$ is
the dataset of GAN’s features, $K_{0}$ is the initial cluster number, and the
output $\Theta=\\{(\mathbf{W}^{t},\mathbf{b},\phi^{t})\\}_{t=1}^{K_{0}-1}$
records the SVM and IoU at each merging step.

The initialization stage has two steps. First, K-means is applied to the
datasets. Second, filtering is performed as described above, resulting in the
initial cluster assignments $\mathcal{A}^{0}$. Then, clusters are iteratively
merged for $K_{0}-1$ steps. In step $t$, an SVM is first trained on dataset
$\mathcal{A}^{t-1}$ , which is obtained from step $t-1$. Next, the IoU and
ECoS are calculated as defined in Eq. 4 and Eq. 5. The class with the lowest
IoU, denoted by $p^{t}$, will be merged with the class with the highest ECoS
with it, denoted by $q^{t}$. The process is repeated until only one class is
left. Without additional information, KLiSH cannot identify a proper number of
classes, just like K-means. We can either slide through cluster numbers and
select manually or set up a threshold $\phi^{*}$ and stop when the IoU reaches
the threshold.

##  4 Evaluation of KLiSH

###  4.1 Experiment Setup

#### Pretrained GANs

We experimented on four widely used GANs, PGGAN [20], StyleGAN [33], StyleGAN2
[34], and StyleGAN2-ADA [35]. These GANs were pretrained on various datasets,
including FFHQ [34], CelebAHQ [64], AFHQ [35], MetFace [35], and LSUN [65]
Car, Bedroom and Church split. The pretrained models were obtained from the
official release 111https://github.com/NVlabs/stylegan3.

GANs | layer indices | channels
---|---|---
StyleGAN2-Bedroom | 9, 11, 13 | 896
StyleGAN2-Church
PGGAN-[Bedroom, Church] | 7, 9, 11, 13 | 960
StyleGAN-[Bedroom, Church]
StyleGAN2-Car | 9, 11, 13, 15 | 960
ADA-[Cat, Dog, Wildlife]
PGGAN-CelebAHQ | 9, 11, 13, 15, 17 | 496
StyleGAN-CelebAHQ
StyleGAN2-FFHQ | 9, 11, 13, 15, 17 | 992
ADA-MetFace
TABLE II: The indices of GAN layers selected for clustering.

#### GAN feature selection

We only collected features from a subset of GAN’s layers mainly due to GPU
memory constraints. To select the layers, we scanned through a generator’s
layers from high resolution to low and stopped adding new layers if the
accumulated number of channels exceeded 1,000. If multiple layers were having
the same resolution, only the last layer of that resolution was selected. The
selected layers for all GANs are shown in Tab. II. The feature maps from the
selected layers are then bilinearly interpolated into the same resolution and
concatenated along the channel axis. For clustering, we use $256\times 256$
resolution. For generating the segmentation (in UFGS and USCS), we use the
full image resolution.

#### Semantic segmentation networks

For face images (CelebAHQ and FFHQ), we trained a DeepLabV3 model with
ResNet50 [66] backbone on CelebAMask-HQ [12]. Following [7], we merged
duplicate classes like “left eye” and “right eye”, resulting in 15 classes.
The model was trained on the training split of CelebAMask-HQ for 20 epochs. We
used Adam with learning rate $10^{-3}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$ and
batch size 32. The learning rate was lowered to $10^{-4}$ after 10 epochs. We
found that 20 epochs were sufficient for the convergence of the DeepLabV3
model. For bedroom and church images, we used a DeepLabV3 model
222https://github.com/zhanghang1989/PyTorch-Encoding pretrained on ADE20K
[67].

#### Baseline clustering algorithms

We compared KLiSH with K-means, AHC, and KASP [19]. K-means and AHC have been
used to cluster GAN’s features [16, 58, 18, 17], but they implicitly require
the centroid assumption (described in Tab. I). Spectral clustering does not
have such a constraint, however, it is not practical for our task as it has an
$O(N^{3})$ time complexity [19]. Therefore, we consider using KASP, a fast
approximate algorithm for spectral clustering. Interestingly, KASP also
performs K-means first and then merges the initial clusters. Different from
KLiSH, KASP uses spectral clustering on the initial centroids to merge them
and does not use linear separability.

We implemented a multi-GPU version of K-means++. KASP was implemented
following the original paper [19]. For AHC, we made two modifications based on
the scikit-learn implementation 333https://scikit-
learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html.
First, we trained a linear classifier on the cluster assignments obtained by
AHC for prediction; otherwise, AHC would rebuild the whole merging tree to
predict new data, which was impractical. Second, we gave AHC less number of
features due to its expensive memory cost. More optimization of AHC might be
possible, but would be beyond the scope of this work.

#### Evaluation of clustering

For each GAN, we sampled a total of 256 images, resulting in a block of
features with shape $(256,256,256,D)$. The only exception is AHC, where we
used 16 images and $64\times 64$ resolution, due to the expensive memory cost
mentioned above. Then, we applied KLiSH, K-means, AHC, and KASP to the sampled
features and evaluated them on the same set of 10k sampled images (different
from those used for clustering).

For evaluation, we consider two widely used metrics, Adjusted Mutual
Information (AMI) and Adjusted Rand Index (ARI). Both AMI and ARI are designed
for general-purpose clustering tasks and may not reflect the performance of
segmentation tasks faithfully. In the context of (supervised) semantic
segmentation, mIoU is the dominant metric. Therefore, we also propose the
Maximum matching mIoU (MIoU) metric defined below,

$\displaystyle\displaystyle\text{MIoU}(\\{\mathcal{Y}_{m}\\}_{m=1}^{M},\\{\mathcal{C}_{k}\\}_{k=1}^{K})$
$\displaystyle=\max_{\mathbf{a}\in\\{0,\ldots,M\\}^{K}}\frac{1}{M}J(\mathbf{a}),$
(6) $\displaystyle J(\mathbf{a};\\{\mathcal{Y}_{m}\\},\\{\mathcal{C}_{k}\\})$
$\displaystyle=\sum_{m=1}^{M}\text{IoU}(\mathcal{Y}_{m},\bigcup_{\\{k|\mathbf{a}_{k}=m\\}}\mathcal{C}_{k}),$
(7)

where $M$ is the number of groundtruth classes, $K$ is the number of clusters,
$\mathcal{Y}_{m}$ is the groundtruth class $m$, and $\mathcal{C}_{r}$ is the
predicted cluster $r$. $\mathbf{a}$ is a $K$-dimensional vector that encodes
the matched class for each cluster. If the $r$-th element of $\mathbf{a}$,
$\mathbf{a}_{r}$ equals 0, then the cluster $r$ does not match any groundtruth
class. Otherwise, $\mathbf{a}_{r}>0$ indicates cluster $r$ is matched to
groundtruth class $\mathbf{a}_{r}$. In short, MIoU measures the segmentation
performance of the predicted clusters after they have been matched correctly
to the groundtruth classes. MIoU should only be compared when two clustering
algorithms produce the same number of clusters, _i.e_., when the cost of label
permutation is identical. This constraint is also required by other clustering
metrics such as AMI and ARI.

Figure 4: Demonstration of MIoU-permuted labels.

The MIoU is calculated by a greedy algorithm as described in Algorithm 2, in
which $(\mathbf{a}|_{\mathbf{a}_{k}\leftarrow m})$ denotes that the
$k^{\text{th}}$ element of $\mathbf{a}$ is set to $m$. To demonstrate the
effectiveness of MIoU, we tested Algorithm 2 on a few images. As shown in Fig.
4, the label permutations found by Algorithm 2 well matched the predicted
clusters to the groundtruth classes.

Input: $\\{\mathcal{Y}_{m}\\}_{m=1}^{M},\\{\mathcal{C}_{k}\\}_{k=1}^{K}$

Output: MIoU, $\mathbf{a}^{t}\in\\{0,1,\ldots,M\\}^{K}$

$\mathbf{a}^{0}=\mathbf{0}$

for  _$t=1,\ldots,K$_ do

$\mathcal{I}^{t}=\\{k|\mathbf{a}_{k}^{t-1}=0\\}\times\\{1,\ldots,M\\}$

$\mathcal{G}^{t}=\\{\mathbf{a}^{\prime}|\mathbf{a}^{\prime}=(\mathbf{a}^{t-1}|_{\mathbf{a}_{k}^{t-1}\leftarrow
m}),(k,m)\in\mathcal{I}^{t}\\}$

$\mathbf{a}^{t}=\max_{\mathbf{a}^{\prime}\in\mathcal{G}^{t}}J(\mathbf{a}^{\prime};\\{\mathcal{Y}_{m}\\},\\{\mathcal{C}_{k}\\})$

MIoU = $J(\mathbf{a}^{t};\\{\mathcal{Y}_{m}\\},\\{\mathcal{C}_{k}\\})$

Algorithm 2 Maximum matching mIoU.

All experiments were conducted using PyTorch [68]. KLiSH required 8 GeForce
RTX 2080 Ti GPUs to run and took less than 30 minutes per GAN model.

Figure 5: Results of K-means, AHC, KASP, and KLiSH on face images and car images. The first row shows the generated images and the other rows show the clustering results obtained with different cluster numbers, denoted by the number after “@”. “final” refers to the selected number of clusters for downstream tasks, which is 26, 11, 30, and 26 from left to right, respectively.  GAN | StyleGAN2 | StyleGAN | PGGAN
---|---|---|---
Dataset | FFHQ | Bedroom | Church | CelebAHQ | Bedroom | Church | CelebAHQ | Bedroom | Church
SVM | 81.7 | 43.5 | 33.7 | 72.2 | 30.9 | 31.4 | 66.4 | 27.7 | 45.4
LSE | 82.7 | 45.2 | 35.2 | 73.8 | 37.3 | 34.8 | 69.3 | 31.1 | 47.8
$\Delta\%$ | -1.3 | -3.8 | -4.3 | -2.2 | -9.8 | -17.2 | -4.2 | -10.9 | -5.0
TABLE III: The semantic extraction performance (in mIoU%) of SVM and LSE on
various GANs and datasets. $\Delta$ denotes the relative difference,
$\Delta=\frac{\text{SVM}-\text{LSE}}{\text{LSE}}$.

###  4.2 Binary Linear Separability Results

As described in Sec. 3.1, KLiSH relies on the binary linear separability of
GAN’s features (Definition 1). To test if this assumption is satisfied, we
compared the performance of semantic segmentation using GAN’s features using
either the SVM or the LSE. The LSE shared the same architecture with the SVM
but was trained with cross-entropy loss, reflecting the classical linear
separability. For each GAN, we sampled 50k images, collected their features,
and obtained their segmentation masks using the pretrained DeepLabV3 model.
Next, we trained LSE and SVM on the features and segmentation masks for 1
epoch using the Adam optimizer with $10^{-3}$ learning rate. The SVM used Eq.
3 with $\lambda_{1}=5000$, which was identical to the value used in KLiSH. For
evaluation, we sampled another different 10k images and measured the mIoU
between the segmentation of LSE or SVM and the segmentation of the DeepLabV3
model.

Their performance is presented in Tab. III. SVM achieved close performance
with LSE in most cases. The relative differences between SVM and LSE were
within 5% on 6 models. Therefore, in general, the classical linear
separability of GAN’s features can be relaxed to the binary linear
separability.

K | metric | K-means | KASP | AHC | KLiSH (ours)
---|---|---|---|---|---
euclidean | euclidean | ward | arccos | euclidean
StyleGAN2-FFHQ
20 | AMI | 64.8 | 61.9 | 62.9 | 65.2 | 70.6
ARI | 57.1 | 54.4 | 60.7 | 65.1 | 70.5
MIoU | 50.2 | 40.3 | 46.4 | 37.7 | 52.7
$26^{\dagger}$ | AMI | 56.3 | 59.1 | 58.0 | 68.0 | 68.3
ARI | 25.1 | 37.2 | 37.7 | 69.2 | 65.4
MIoU | 52.6 | 48.9 | 46.9 | 43.8 | 56.3
30 | AMI | 56.3 | 59.3 | 57.7 | 65.9 | 67.6
ARI | 25.1 | 38.5 | 35.7 | 64.2 | 64.6
MIoU | 52.6 | 53.2 | 49.9 | 45.5 | 58.0
StyleGAN-CelebAHQ
20 | AMI | 53.0 | 55.5 | 53.1 | 62.5 | 69.0
ARI | 38.3 | 38.1 | 49.1 | 67.0 | 73.1
MIoU | 30.9 | 36.8 | 30.5 | 25.4 | 41.5
$30^{\dagger}$ | AMI | 51.9 | 49.7 | 43.4 | 57.9 | 63.6
ARI | 28.2 | 30.5 | 21.4 | 54.8 | 61.8
MIoU | 38.5 | 33.5 | 31.7 | 27.7 | 45.7
PGGAN-CelebAHQ
20 | AMI | 37.9 | 39.3 | 39.4 | 37.1 | 51.0
ARI | 18.5 | 19.4 | 20.5 | 30.1 | 53.5
MIoU | 15.2 | 16.9 | 16.4 | 19.1 | 23.1
$26^{\dagger}$ | AMI | 35.1 | 37.9 | 37.6 | 36.0 | 51.0
ARI | 11.5 | 15.3 | 15.6 | 28.7 | 53.5
MIoU | 17.4 | 19.0 | 16.4 | 19.2 | 29.0
30 | AMI | 35.1 | 36.2 | 38.0 | 38.1 | 50.2
ARI | 11.5 | 12.4 | 15.4 | 30.7 | 52.4
MIoU | 17.4 | 18.6 | 18.4 | 19.6 | 29.8

TABLE IV: Quantitative evaluation of clustering algorithms on face image
generators. The final number of clusters used is indicated by a $\dagger$. For
all metrics, the larger the better. The clustering results are reported as the
best one among the 5 trials.

###  4.3 KLiSH Clustering Results

#### Quantitative evaluations

We evaluated the performance of clustering algorithms on GANs trained on face
images. We report the metrics at cluster numbers K=20, K=30, and the finally
selected number in Tab. IV.

It was clear that KLiSH outperformed existing clustering algorithms. For AMI
and MIoU, KLiSH was superior to all baselines in all cases. Notably, at the
finally selected number of clusters, KLiSH achieved significantly higher MIoU
scores. It indicated that KLiSH achieved the best segmentation performance at
the cluster number that would be used by downstream tasks. For ARI, KLiSH also
achieved the highest score in almost every case, except for K=26 in
StyleGAN2-FFHQ, where AHC (arccos) achieved a better score. Even though,
qualitative results showed that AHC was not better than KLiSH in this case
(see row “final” of column “StyleGAN2-FFHQ” of Fig. 5). AHC failed to cluster
“eye”, “nose”, “upper lip”, “lower lip”, but KLiSH successfully identified all
those clusters. KLiSH’s lower ARI might result from leaving two clusters for
“face” and “neck”. However, in this case, it would be rather easy to merge
them manually. In contrast, dividing the undersegmented clusters obtained by
AHC would be more difficult.

In summary, KLiSH was better than K-means, AHC, and KASP on GAN’s features.

#### Qualitative results

We visualize the clusters obtained by all the cluster algorithms on various
GANs and datasets in Fig. 5. The results were from the same trial (best out of
5 trials) as in Tab. IV. We had two main observations.

First, KLiSH was significantly more accurate than K-means, AHC, and KASP. In
StyleGAN2-FFHQ, K-means@30 failed to cluster “upper lip”, “lower lip”,
“teeth”, “brow” classes and still undersegmented “background” and “hair”.
AHC@30 already failed to cluster “nose” and the classes in mouth region. In
contrast, KLiSH@30 had no undersegmentation of groundtruth classes. Similarly,
for car images, KLiSH@final successfully segmented the “headlight”, “wheel”,
and even two “side glasses”. In contrast, the most competitive baseline,
AHC@final not only undersegmented “background” and “wheel” classes, but also
completely missed the “headlight”.

Second, the advantage of KLiSH over existing clustering algorithms was
consistent across GANs and datasets. For StyleGAN-CelebAHQ, AHC@final missed
“nose”, “lip”, and “teeth” classes and undersegmented “hair”, while
KLiSH@final did not. For PGGAN-CelebAHQ, although all the clustering
algorithms did not obtain good results – probably because PGGAN had learned
the dataset well. Still, KLiSH@final successfully segmented “ear” and “hair”
classes, but AHC@final missed “ear” and mixed “hair” with “background” (see
the rightmost column). We provide more visualization in Fig. 7, which is also
supportive of the observations described above.

Figure 6: Demonstration of a merging step. The first two columns show the
images and clusters. The confidence maps
$\\{\mathbf{s}_{k}^{\dagger}\\}_{k=1}^{M}$ of the image are visualized
(partially) in other columns. $\\{\mathbf{s}_{k}^{\dagger}\\}_{k=1}^{M}$ take
value within [0, 1] and are visualized using the color bar on the right. The
cluster in yellow box (say cluster $p$) has the smallest IoU (Eq. 4) with
K=35. The cluster in green box (say cluster $q$) has the largest ECoS (Eq. 5)
with cluster $p$. Cluster $p$ and $q$ are then merged in this step.  Figure 7:
Comparison of clustering algorithms on various datasets and GANs. “final”
refers to 18, 9, 16, 15, and 28 from left to right.

#### Visualization of a merging step

To help readers gain a concrete picture of how KLiSH merged the clusters, we
also visualized a single merging step in Fig. 6. See the cluster confidence
map of “nose” and “brow” (Rows 2 and 3, Column 4 in the right figure), the red
regions are dominated by deep red, indicating that the SVM predicts them with
high confidence. On the contrary, the heatmap denoted by the yellow box has a
large proportion of shallow red, indicating that the confidence of SVM to
predict that cluster was low. This cluster had the lowest IoU at K=35 and was
subsequently merged with another oversegmented cluster of hair, resulting in
better cluster assignments.

##  5 Applications

Method | #shots | #classes | mIoU(%)
---|---|---|---
UFGS (KLiSH) | 0 | 15 | 47.2
UFGS (AHC) | 0 | 15 | 36.6
LSE[7] | 1 | 15 | 47.1 $\pm$ 2.8
10 | 58.5 $\pm$ 0.9
RepurposeGAN [15] | 10 | 9 | 68.0
DatasetGAN [14] | 16 | 8 | 70.0
TABLE V: The performance of two UFGS methods and several few-shot learning
methods.

method | #samples | skin | nose | eye-g | eye | brow | ear | teeth | u-lip | l-lip | hair | hat | ear-r | neck | cloth | mouth*
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
UFGS (KLiSH) | (K=26) | 82.2 | 76.0 | 25.7 | 50.4 | 60.2 | 50.3 | 32.1 | 52.2 | 51.0 | 76.5 | 0.0 | 0.0 | 70.1 | 34.0 | -
UFGS (AHC) | (K=26) | 62.0 | 19.3 | 27.6 | 62.6 | 30.6 | 49.3 | 0.00 | 47.1 | 60.3 | 80.5 | 0.1 | 11.0 | 34.6 | 27.2 | -
LSE | N=1 | 83.8 | 81.7 | 1.9 | 72.9 | 51.0 | 45.6 | 32.7 | 64.8 | 68.2 | 69.0 | 0.6 | 1.7 | 68.8 | 16.1 | -
N=10 | 89.3 | 85.5 | 25.7 | 72.2 | 61.4 | 58.9 | 60.5 | 70.1 | 74.2 | 85.2 | 6.3 | 11.5 | 74.7 | 43.6 | -
RepurposeGAN | N=1 | - | 76 | - | 58 | - | - | - | - | - | - | - | - | - | - | 71
N=10 | 90 | 85 | - | 75 | 68 | 37 | - | - | - | 84 | - | - | 73 | 16 | 87

TABLE VI: IoU(%) per class of DeepLabV3 trained on synthetic datasets
generated with different annotation methods. The evaluation is conducted on
real images from the test split of CelebAMask-HQ. “eye-g”, “u-lip”, “l-lip”
denotes “eyeglasses”, “upper lip”, and “lower lip”, respectively. The mouth
class is a union of “teeth”, “u-lip”, and “l-lip”. Our UFGS achieved close
performance with one-shot LSE on most classes and outperformed one-shot LSE on
hard classes like “eyeglasses” and “cloth”.

Figure 8: UFGS results on _real_ images from CelebAMask-HQ (rows 1 and 2) and
PASCAL VOC (rows 3 and 4). Figure 9: The qualitative results of USCS on
various GANs and datasets. The semantic masks are drawn by users and the
modifications on the masks are indicated by dashed boxes. The image on the
right of each mask is generated by our USCS model.

###  5.1 Unsupervised Fine-Grained Segmentation

Semantic segmentation labeling is very expensive. As described by Lee _et al_.
in CelebAMask-HQ [12], annotating fine-grained semantic segmentation is not
only time-consuming but also involves multiple iterations of refinement. To
reduce the labeling cost, we propose an approach for Unsupervised Fine-grained
Segmentation (UFGS).

Existing methods on unsupervised segmentation [52, 54, 55, 56] only segment
foreground from background, thus are not applicable to UFGS. A more closely
related task to UFGS is few-shot learning-based segmentation [7, 15, 14].
Although they reduce the number of annotations, the cost of fine-grained
semantic annotations is still significant. Zhang _et al_. [14] reports that
annotating a single face image cost on average 20 minutes _per image_. In
contrast, the only labeling cost involved in UFGS is naming and merging a few
labels, which could take less than one minute for a _whole dataset_.

#### Method

The proposed method for UFGS is as follows: First, train a GAN on the target
dataset, run KLiSH on the GAN, and select a proper number of semantic
clusters. Next, generate an image-segmentation dataset by sampling images from
the GAN and using the SVM returned by KLiSH to obtain the segmentation masks
of the image. We apply the SVM directly on the feature maps upsampled to the
resolution of images. Then, train a segmentation network, _e.g_., DeepLabV3
[11], on the synthetic dataset. Finally, segment _real_ images in the original
dataset.

#### Evaluation

We evaluated the performance of our UFGS method on face images using the test
split of the CelebAMask-HQ dataset. The cluster assignments were matched to
the ground-truth classes by maximizing MIoU (Algorithm 2) using 100 images
from the validation split. Note that the images and labels are not used to
train the model. In practice, users can match clusters to labels with just
several clicks. The performance of our method was compared to few-shot
learning methods [7, 15, 14].

To the best of our knowledge, except for face images, there are no large-scale
semantic annotations on the datasets described in Sec. 4.1. PASCAL VOC [69]
contains car images, but they have a significant domain gap with the car
images from the GAN’s dataset, LSUN-Car. In two relevant works, DatasetGAN[14]
and RepurposeGAN [15], the authors make their own test sets by filtering
PASCAL VOC and have not released them as of now. Therefore, we only present
qualitative results on PASCAL VOC and report quantitative results on
CelebAMask-HQ.

#### Experiment setup

We used StyleGAN2-FFHQ and StyleGAN2-Car for UFGS on CelebAMask-HQ and PASCAL
VOC cars, respectively. The baseline for comparison was UFGS with AHC, as it
was the most competitive method with KLiSH on StyleGAN2-FFHQ according to Tab.
IV. The settings were identical to UFGS with KLiSH except that the datasets
were generated with clusters found by AHC. We also compared UFGS with KLiSH to
a few-shot semantic extractor, LSE. We reproduced LSE on StyleGAN2-FFHQ and
StyleGAN2-Car following the settings in [7]. In brief, we randomly sampled 1
or 10 images (for one-shot or 10-shot experiments) and obtained their
segmentation using the pretrained DeepLabV3 model. Then, we trained the LSE on
the features and labels of those images for 6,400 iterations. To address the
variation of few-shot experiments, we trained LSE five times, each time with a
different set of image annotations.

After training the LSE and obtaining the KLiSH clusters, we then compared
their performances on benefiting the downstream segmentation task. We first
sampled a synthesized image segmentation dataset for each of the LSE models
and KLiSH results, which consisted of 50k images and segmentation masks. Then,
we trained a DeepLabV3 model on each dataset following the training settings
described in Sec. 4.1. Finally, we tested the DeepLabV3 models on the real
images from the test split of the CelebAMask-HQ dataset.

#### Results

We present the qualitative results of our UFGS method in Fig. 8. As shown in
Row 1 and 2, UFGS model segmented facial classes like “eye”, “brow”, “lips”,
“teeth” and “nose” well. For car images in Row 3 and 4, our UFGS method also
successfully segmented “headlight”, “body”, “glasses”, and “wheel”. However,
we found there was a domain gap between synthesized images and real images.
See Row 1, Col 3 and 4 in Fig. 8. The hand in front of the face is a rare case
in the dataset. Therefore, the segmentation model cannot deal with this case
well.

The quantitative results are reported in Tab. V and Tab. VI. As shown in Tab.
V, KLiSH surpassed UFGS with AHC by a large margin. In comparison with few-
shot learning methods, UFGS achieved close performance with the one-shot
method, LSE, but lagged behind the 10-shot methods.

In summary, though the performance of our UFGS method was limited by the
domain gap as a result of using GANs, we for the first time showed accurate
fine-grained semantic segmentation in an unsupervised learning setting.

###  5.2 Unsupervised Semantic-Conditional Synthesis

Enhancing controllability over the generated images is important for
commercial applications of GANs. Most GANs do not allow the user to control
the spatial structure of generated images precisely. Semantic-Controllable
Synthesis (SCS) aims to provide controllability over precise spatial
structures of generated images. Though achieving greater controllability for
image synthesis, existing methods for SCS, _e.g_., pix2pix [42], SEAN [45] and
GauGAN [44], rely heavily on large-scale human annotated semantic masks.

Recently, the labeling requirements of SCS have been lowered to a few
annotations [7, 13]. Yet, the user still has to label a few multi-class fine-
grained segmentation masks, which limits the application in practice. For
example, users may want to enable SCS on images of tigers. If there are no
semantic segmentation masks on these images, the user needs to annotate 16 to
40 images. The annotation process would take around 2 to 5 hours, according to
Zhang _et al_. [14].

To lower the label dependency of SCS methods, we propose the task of
Unsupervised SCS, which aims to achieve semantic-controllable image synthesis
on arbitrary datasets without human annotations.

#### Method

Our USCS application works as follows. First, the user shall provide a target
image dataset. Then, we train an unconditional GAN and generate an image-
segmentation dataset using the same method as described in Sec. 5.1. Finally,
we train a GauGAN [44] on the generated dataset. The above process can be done
offline. Then users can draw and edit on the semantic mask and the GauGAN will
generate desired images.

We developed a web-based interface to help users generate images with desired
structures. Before the user starts painting, a few images and their semantic
masks are shown as examples. Next, the user can click on a palette and paint
with selected semantic classes. Then, the user can draw freely and submit the
painting to the server, which will return an image synthesized given the
semantic mask.

#### Experiment setup

We conducted the USCS experiment on StyleGAN2 models trained on FFHQ and Car
datasets and StyleGAN2-ADA models trained on MetFaces, AFHQ [40] Cat, Dog, and
Wildlife split. Similar to the setting of UFGS, we also synthesized a dataset
with 50k paired images and segmentation masks from each GAN and trained a
GauGAN on each dataset. The GauGAN was trained using the official release
444https://www.github.com/nvlabs/spade for 10 epochs on 8 GPUs. The training
took around 100 hours for a GauGAN model.

#### Results

The editing results on different datasets and GAN models are presented in Fig.
9. We observed that USCS supported precise, diverse, and even out-of-
distribution image editing operations. First, USCS respected the semantic mask
well, accurately translating the user’s semantic input into corresponding
images while maintaining good photo-realisticity. See the “big round eye”
example in ADA-Cat. The enlarged eyes had a matched shape as specified by the
user. Second, our USCS method well demonstrated the diversity and degree of
freedom of editing operations. As shown in the left part of StyleGAN2-FFHQ,
the shape of the sunglasses could even be changed to triangle. Third, the out-
of-distribution editing operations could also lead to photorealistic synthesis
results. See the “elf ear” examples in StyleGAN2-FFHQ and ADA-Wild, the
synthesized sharp ear is not present in the training set of the model.
Besides, the “blink eye” in ADA-MetFace and ADA-Wild, and “more wheels” in
StyleGAN2-Car are all out-of-distribution semantic masks.

To the best of our knowledge, SCS cannot be applied to datasets of wildlife
and artistic portraits, _etc_., because so far no semantic annotations on such
datasets are available. EditGAN [13] relies on 16 to 30 fine-grained
segmentation masks to enable semantic editing on Cat, Dog, _etc_., but the
cost of labeling is still high (around 10 minutes per image). In contrast, the
USCS model is free from the constraint of annotations. Please find a
demonstration of our USCS web application in Supplementary Video 1 and have a
try on our online demonstration555https://atlantixjj.github.io/KLiSH/.

##  6 Limitations

Our method is generally constrained by the performance of GANs. First, the
clustering performance of KLiSH is dependent on the degree of linear
separability of GAN’s features. If the GAN does not encode semantics into _one
vs. rest_ linearly separable clusters, then the KLiSH might fail to find
meaningful clusters. Second, the performance of UFGS and USCS is limited by
the synthesis quality of the original GAN. When the GAN fails to generate
realistic images, the UFGS model will have a domain gap when applied to real
images and USCS will also generate unrealistic images.

##  7 Conclusion

In this paper, we propose KLiSH to find semantic clusters in GAN’s features.
By making use of GAN’s linear separability in semantics, KLiSH achieves better
clustering results than conventional clustering algorithms including K-means,
AHC, and KASP. The clusters found by KLiSH accurately correspond to the
semantic parts of generated images. Therefore, for each pretrained GAN, we can
generate an image-segmentation dataset for downstream tasks. One can easily
enable semantics-based applications including UFGS and USCS with the dataset
generated by GANs. The surprising performance of UFGS and USCS make us believe
that GANs can play a more important role in representation learning.

## References

  * [1] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” _PAMI_ , vol. 35, no. 8, pp. 1798–1828, 2013.
  * [2] K. H. R. Chan, Y. Yu, C. You, H. Qi, J. Wright, and Y. Ma, “Redunet: A white-box deep network from the principle of maximizing rate reduction,” _JMLR_ , vol. 23, no. 114, pp. 1–103, 2022.
  * [3] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _NeurIPS_ , 2014, pp. 2672–2680.
  * [4] Y. Shen, C. Yang, X. Tang, and B. Zhou, “Interfacegan: Interpreting the disentangled face representation learned by gans,” _PAMI_ , vol. 44, no. 4, pp. 2004–2018, 2022.
  * [5] C. Yang, Y. Shen, and B. Zhou, “Semantic hierarchy emerges in deep generative representations for scene synthesis,” _IJCV_ , vol. 129, no. 5, pp. 1451–1466, 2021.
  * [6] D. Bau, J.-Y. Zhu, H. Strobelt, B. Zhou, J. B. Tenenbaum, W. T. Freeman, and A. Torralba, “Visualizing and understanding generative adversarial networks,” in _ICLR_ , 2019.
  * [7] J. Xu and C. Zheng, “Linear semantics in generative adversarial networks,” in _CVPR_ , 2021, pp. 9351–9360.
  * [8] A. Jahanian, L. Chai, and P. Isola, “On the” steerability” of generative adversarial networks,” in _ICLR_ , 2019.
  * [9] A. Voynov and A. Babenko, “Unsupervised discovery of interpretable directions in the gan latent space,” in _ICML_. PMLR, 2020, pp. 9786–9796.
  * [10] Y. Shen and B. Zhou, “Closed-form factorization of latent semantics in gans,” in _CVPR_ , 2021, pp. 1532–1540.
  * [11] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” _arXiv preprint arXiv:1706.05587_ , 2017.
  * [12] C.-H. Lee, Z. Liu, L. Wu, and P. Luo, “Maskgan: Towards diverse and interactive facial image manipulation,” in _CVPR_ , 2020, pp. 5549–5558.
  * [13] H. Ling, K. Kreis, D. Li, S. W. Kim, A. Torralba, and S. Fidler, “Editgan: High-precision semantic image editing,” in _NeurIPS_ , 2021.
  * [14] Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler, “Datasetgan: Efficient labeled data factory with minimal human effort,” in _CVPR_ , 2021, pp. 10 145–10 155.
  * [15] N. Tritrong, P. Rewatbowornwong, and S. Suwajanakorn, “Repurposing gans for one-shot semantic part segmentation,” in _CVPR_ , 2021, pp. 4475–4485.
  * [16] E. Collins, R. Bala, B. Price, and S. Süsstrunk, “Editing in style: Uncovering the local semantics of GANs,” in _CVPR_ , 2020.
  * [17] O. Kafri, O. Patashnik, Y. Alaluf, and D. Cohen-Or, “Stylefusion: Disentangling spatial segments in stylegan-generated images,” _TOG_ , 2022\.
  * [18] Y. Zheng, Y.-K. Huang, R. Tao, Z. Shen, and M. Savvides, “Unsupervised disentanglement of linear-encoded facial semantics,” in _CVPR_ , 2021, pp. 3917–3926.
  * [19] D. Yan, L. Huang, and M. I. Jordan, “Fast approximate spectral clustering,” in _Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , ser. KDD ’09. New York, NY, USA: Association for Computing Machinery, 2009, p. 907–916.
  * [20] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” _arXiv preprint arXiv:1710.10196_ , 2017.
  * [21] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in _NeurIPS_ , 2017, pp. 6626–6637.
  * [22] E. L. Denton, S. Chintala, R. Fergus _et al._ , “Deep generative image models using a laplacian pyramid of adversarial networks,” in _NeurIPS_ , 2015, pp. 1486–1494.
  * [23] A. Odena, C. Olah, and J. Shlens, “Conditional image synthesis with auxiliary classifier gans,” in _ICML_. PMLR, 2017, pp. 2642–2651.
  * [24] X. Huang, Y. Li, O. Poursaeed, J. Hopcroft, and S. Belongie, “Stacked generative adversarial networks,” in _CVPR_ , 2017, pp. 5077–5086.
  * [25] T. R. Shaham, T. Dekel, and T. Michaeli, “Singan: Learning a generative model from a single natural image,” in _CVPR_ , 2019, pp. 4570–4580.
  * [26] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in _ICML_. PMLR, 2017, pp. 214–223.
  * [27] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, “Improved training of wasserstein gans,” in _NeurIPS_ , 2017, pp. 5767–5777.
  * [28] N. Kodali, J. Abernethy, J. Hays, and Z. Kira, “On convergence and stability of gans,” _arXiv preprint arXiv:1705.07215_ , 2017.
  * [29] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in _NeurIPS_ , 2016, pp. 2180–2188.
  * [30] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _arXiv preprint arXiv:1511.06434_ , 2015.
  * [31] J. Zhao, M. Mathieu, and Y. LeCun, “Energy-based generative adversarial networks,” in _ICLR_ , 2017.
  * [32] D. Berthelot, T. Schumm, and L. Metz, “Began: Boundary equilibrium generative adversarial networks,” _arXiv preprint arXiv:1703.10717_ , 2017.
  * [33] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _CVPR_ , 2019, pp. 4401–4410.
  * [34] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in _CVPR_ , 2020, pp. 8110–8119.
  * [35] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, “Training generative adversarial networks with limited data,” _arXiv preprint arXiv:2006.06676_ , 2020.
  * [36] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila, “Alias-free generative adversarial networks,” _arXiv preprint arXiv:2106.12423_ , 2021.
  * [37] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in _ICML_. PMLR, 2019, pp. 7354–7363.
  * [38] A. Brock, J. Donahue, and K. Simonyan, “Large scale gan training for high fidelity natural image synthesis,” in _ICLR_ , 2018.
  * [39] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation,” in _CVPR_ , 2018, pp. 8789–8797.
  * [40] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, “Stargan v2: Diverse image synthesis for multiple domains,” in _CVPR_ , 2020, pp. 8188–8197.
  * [41] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _CVPR_ , 2017, pp. 2223–2232.
  * [42] P. Isola, J. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _CVPR_ , 2017, pp. 5967–5976.
  * [43] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” in _CVPR_ , 2018, pp. 8798–8807.
  * [44] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Gaugan: semantic image synthesis with spatially adaptive normalization,” in _ACM SIGGRAPH 2019 Real-Time Live!_ , 2019, pp. 1–1.
  * [45] P. Zhu, R. Abdal, Y. Qin, and P. Wonka, “Sean: Image synthesis with semantic region-adaptive normalization,” in _CVPR_ , 2020, pp. 5104–5113.
  * [46] Y. Jo and J. Park, “Sc-fegan: Face editing generative adversarial network with user’s sketch and color,” in _CVPR_ , 2019, pp. 1745–1753.
  * [47] Y. Shi, X. Yang, Y. Wan, and X. Shen, “Semanticstylegan: Learning compositional generative priors for controllable image synthesis and editing,” in _CVPR_ , 2022, pp. 11 254–11 264.
  * [48] W. Sun and T. Wu, “Learning layout and style reconfigurable gans for controllable image synthesis,” _PAMI_ , pp. 1–1, 2021.
  * [49] D. Bau, S. Liu, T. Wang, J.-Y. Zhu, and A. Torralba, “Rewriting a deep generative model,” in _ECCV_ , 2020.
  * [50] S.-Y. Wang, D. Bau, and J.-Y. Zhu, “Sketch your own gan,” in _CVPR_ , 2021, pp. 14 050–14 060.
  * [51] D. Galeev, K. Sofiiuk, D. Rukhovich, M. Romanov, O. Barinova, and A. Konushin, “Learning high-resolution domain-specific representations with a gan generator,” in _Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR)_. Springer, 2021, pp. 108–118.
  * [52] Y. Benny and L. Wolf, “Onegan: Simultaneous unsupervised learning of conditional image generation, foreground segmentation, and fine-grained clustering,” in _ECCV (26)_ , 2019, pp. 514–530.
  * [53] K. K. Singh, U. Ojha, and Y. J. Lee, “Finegan: Unsupervised hierarchical disentanglement for fine-grained object generation and discovery,” in _CVPR_ , 2019, pp. 6490–6499.
  * [54] R. Abdal, P. Zhu, N. J. Mitra, and P. Wonka, “Labels4free: Unsupervised segmentation using stylegan,” in _CVPR_ , 2021, pp. 13 970–13 979.
  * [55] L. Melas-Kyriazi, C. Rupprecht, I. Laina, and A. Vedaldi, “Finding an unsupervised image segmenter in each of your deep generative models,” in _ICLR_ , 2021.
  * [56] Y. Yang, H. Bilen, Q. Zou, W. Y. Cheung, and X. Ji, “Unsupervised foreground-background segmentation with equivariant layered gans.” _arXiv preprint arXiv:2104.00483_ , 2021.
  * [57] A. Voynov, S. Morozov, and A. Babenko, “Object segmentation without labels with large-scale generative models,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 10 596–10 606.
  * [58] Y. He, Z. Zhang, J. Zhu, Y. Shen, and Q. Chen, “Interpreting class conditional gans with channel awareness,” _arXiv preprint arXiv:2203.11173_ , 2022.
  * [59] S. Gao, Z.-Y. Li, M.-H. Yang, M.-M. Cheng, J. Han, and P. H. S. Torr, “Large-scale unsupervised semantic segmentation.” _CVPR_ , 2021\.
  * [60] W. V. Gansbeke, S. Vandenhende, S. Georgoulis, and L. V. Gool, “Unsupervised semantic segmentation by contrasting object mask proposals,” in _CVPR_ , 2021, pp. 10 052–10 062.
  * [61] S. Liu, L. Zhang, X. Yang, H. Su, and J. Zhu, “Unsupervised part segmentation through disentangling appearance and shape,” in _CVPR_ , 2021, pp. 8355–8364.
  * [62] R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, “Liblinear: A library for large linear classification,” _JMLR_ , vol. 9, no. Aug, pp. 1871–1874, 2008.
  * [63] D. C. Liu and J. Nocedal, “On the limited memory bfgs method for large scale optimization,” _Mathematical programming_ , vol. 45, no. 1, pp. 503–528, 1989.
  * [64] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _ICCV_ , December 2015.
  * [65] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” _arXiv preprint arXiv:1506.03365_ , 2015.
  * [66] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016, pp. 770–778.
  * [67] B. Zhou, H. Zhao, F. X. P. Fernandez, S. Fidler, and A. Torralba, “Scene parsing through ade20k dataset,” in _CVPR_ , 2017.
  * [68] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _NeurIPS_. Curran Associates, Inc., 2019, pp. 8024–8035.
  * [69] M. Everingham, L. Van Gool, C. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _IJCV_ , vol. 88, pp. 303–338, 06 2010.

|  Jianjin Xu (S’01-M’08-SM’13) received a B.E. degree in computer science
from the Tsinghua University, Beijing, China, in 2019, and a M.S. degree in
computer science from the Columbia University, New York City, NY, U.S., in
2021. He is currently an Assistant Research Scientist at the School of
Mathematics and Computer Science, Panzhihua University, Sichuan, China. His
current research interests include generative models, neural network
interpretation, and computer vision. Previously he was a research assistant at
the Department of Computer Science and Technology, Tsinghua University.
---|---
|  Zhaoxiang Zhang received his bachelor’s degree in Circuits and Systems from
the University of Science and Technology of China (USTC) in 2004. In 2004, he
joined the National Laboratory of Pattern Recognition (NLPR), Institute of
Automation, Chinese Academy of Sciences, under the supervision of Professor
Tieniu Tan, and he received his Ph.D. degree in 2009. In October 2009, he
joined the School of Computer Science and Engineering, Beihang University, as
an Assistant Professor (2009-2011), an Associate professor (2012-2015) and the
vise-director of the Department of Computer application technology
(2014-2015). In July 2015, he returned to the Institute of Automation, Chinese
Academy of Sciences. He is now a full Professor in the Center for Research on
Intelligent Perception and Computing (CRIPAC) and the National Laboratory of
Pattern Recognition (NLPR). His research interests include Computer Vision,
Pattern Recognition, and Machine Learning. Recently, he specifically focuses
on biologically inspired intelligent computing and its applications on human
analysis and scene understanding. He has published more than 150 papers in the
international journals and conferences, including reputable international
journals such as IEEE TPAMI, IJCV, JMLR and top level international
conferences like CVPR, ICCV, ECCV, ICLR, NeurIPS, AAAI and IJCAI. He served as
the associated editor of IEEE TCSVT, PR, and Frontiers of Computer Science. He
served as the area chair of international conferences like CVPR, ICCV, AAAI,
IJCAI. He is a senior member of IEEE.
---|---
|  Xiaolin Hu (S’01-M’08-SM’13) received B.E. and M.E. degrees in automotive
engineering from the Wuhan University of Technology, Wuhan, China, in 2001 and
2004, respectively, and a Ph.D. degree in automation and computer-aided
engineering from the Chinese University of Hong Kong, Hong Kong, in 2007. He
is currently an Associate Professor at the Department of Computer Science and
Technology, Tsinghua University, Beijing, China. His current research
interests include deep learning and computational neuroscience. At present, he
is an Associate Editor of the IEEE Transactions on Pattern Analysis and
Machine Intelligence, IEEE Transactions on Image Processing, and Cognitive
Neurodynamics. Previously he was an Associate Editor of the IEEE Transactions
on Neural Networks and Learning Systems.
---|---