# Unsupervised Part Discovery via Feature Alignment

Mengqi Guo1 Yutong Bai2 Zhishuai Zhang2 Adam Kortylewski2 Alan Yuille2
1Beihang University 2Johns Hopkins University
{im.guomengqi, ytongbai, zhshuai.zhang, akortyl1<EMAIL_ADDRESS>

###### Abstract

Understanding objects in terms of their individual parts is important, because
it enables a precise understanding of the objects’ geometrical structure, and
enhances object recognition when the object is seen in a novel pose or under
partial occlusion. However, the manual annotation of parts in large scale
datasets is time consuming and expensive. In this paper, we aim at discovering
object parts in an unsupervised manner, , without ground-truth part or
keypoint annotations. Our approach builds on the intuition that objects of the
same class in a similar pose should have their parts aligned at similar
spatial locations. We exploit the property that neural network features are
largely invariant to nuisance variables and the main remaining source of
variations between images of the same object category is the object pose.
Specifically, given a training image, we find a set of similar images that
show instances of the same object category in the same pose, through an affine
alignment of their corresponding feature maps. The average of the aligned
feature maps serves as pseudo ground-truth annotation for a supervised
training of the deep network backbone. During inference, part detection is
simple and fast, without any extra modules or overheads other than a feed-
forward neural network. Our experiments on several datasets from different
domains verify the effectiveness of the proposed method. For example, we
achieve 37.8 mAP on VehiclePart, which is at least 4.2 better than previous
methods.

##  1 Introduction

Computer vision is to a large extent concerned with the analysis of objects in
images. An important challenge in object analysis is to learn representations
that are robust to partial occlusion, or changes in the object pose and
appearance. Part detectors have emerged as an important intermediate object
representation that facilitates to achieve robustness to these variations. As
a result, part detectors have become of central importance for many visual
understanding tasks, including viewpoint estimation [17], human pose
estimation [3], action recognition [14], feature matching [13], image
classification [30], and 3D reconstruction [9]. While part detectors are very
useful for computer vision, a main limitation is that detectors are typically
learned using large amounts of annotated training data [2, 6, 26, 36].
However, in practice, obtaining such annotations is very time consuming.
Therefore, it is difficult to scale fully supervised approaches to novel
categories, which raises the need for unsupervised learning approaches.

The main challenge for unsupervised approach to learn part detectors is that
they need to learn a set of features/neurons corresponding to semantically
meaningful parts tightly on a single image, and consistently across different
images of the same object class. However, objects in images are highly
variable due to changes in their pose, shape, and texture, as well as due to
changes in the scene illumination, and the background clutter. The same parts
may have slightly different appearance at different images, and it’s non
difficult to learn that without supervision.

Figure 1: Examples demonstrate the intuition of our method. Upper left:
training image set. Lower left: a training image and two similar images, with
distinguishable keypoints matched. Lower middle: the heatmap of an output
channel, which corresponds to the front wheel, but with some false positives.
Lower right: pseudo ground-truth generated from the training image and the
similar images, for different output channels. This could be used to supervise
the output feature maps shown in the lower middle.

In this paper, we propose an unsupervised framework for learning part
detectors. Based on a collection of images of any object type, we learn part
detectors that are consistent across different object instances, 3D poses and
articulations. The key intuition underlying our work is that part detectors
should fire consistently across different images, w.r.t. some spatial
transforms. This should be true especially for those images which are roughly
similar with each other, as shown in Figure 1. This suggests that even without
ground-truth part annotation, there still exists supervisions, coming from the
coherence between similar images. To this end, we propose our training
approach with three steps. 1) Given a training image, find a set of other
training images similar to it, by comparing the CNN embedding of any off-the-
shelf ImageNet-pretrained models. This allows us to enforce coherence of part
detectors easier, without dealing with large transformation among images. 2)
Align the similar images with the training image with some spatial
transformation, by exploiting matched feature pairs (, the features at wheels,
window corners locations are more distinguishable to be matched). The strongly
matched feature pairs can suggest the spatial correspondence between the
training image and the similar images. 3) Finally, we can encourage coherence
between the feature maps (those used for part detection) of the training image
and the similar images. To achieve that, we generate a pseudo ground-truth map
from the average of the aligned feature maps, and use that pseudo ground-truth
map to supervise the feature map of each of the images. We also consider the
intuition that the part detectors should fire sparsely on each image, so we
add some constraints when generating the pseudo map by limiting the number of
locations each feature map channel can correspond to. This can avoid a trivial
solution, and allow the learned part detectors to be diverse and cover as more
actual parts as possible.

Following the common scheme of related work on unsupervised part detection, we
quantitatively evaluate our unsupervised part detection method for several
different datasets on different domains, including VehiclePart [26], CelebA
[12], CUB [25], where we achieve state-of-the-art performance, as well as MAFL
[33], where we perform comparably to related work. Our qualitative results
show that our model predicts part landmarks precisely and consistently even in
different camera viewpoints and object poses.

##  2 Related Work

Recently, with the emergence and development of deep neural networks [11],
object recognition[4, 5, 8, 18] has achieved great success. As a detailed
version of object recognition, part discovery and detection have been
receiving more and more research attentions, for different domains, such as
vehicles [1, 26], faces [34], humans [15]. However, this is still a relatively
less studied task, mainly due to the lack of annotations, which could be
magnitudes of more expensive to obtain than object-level annotations.

Previous works for unsupervised part detection, including [26, 27, 37], show
that parts usually have similar local appearances, can be captured by
individual neurons [37] or k-means clusters [27] of internal feature vectors.
We choose clustering of internal feature vectors as our baseline, by adding
one more layer to the backbone with the cluster center features as the layer
weights. However, different parts and background may have similar local
appearance, which could lead to fuzzy clusters. Other works find that parts
can be supported by surrounding contexts, and can be detected by in voting
scheme [26]. Some researches ( [23]) reformulate the landmark detection
problem into image deformation. Nevertheless, those methods mainly take only
the feature inside individual images or feature maps into consideration, while
ignoring the similarity and consistency across different images or feature
maps. Related work [7] considers the local feature as well as semantic
consistency between images. They build geometric concentration loss to
concentrate the segmentation from fragments and equivalence loss which is
based on the hypothesis that images have same part segmentation predictions
with regard to the corresponding transformation. Then, they also build
semantic consistency loss with a set of semantic part bases, which does not
take spatial alignment into account simultaneously. On the other hand, our
method exploits the spatial correspondence between similar images, from the
intuition that similar images should have the same parts at similar locations.

Since unsupervised part detection is difficult, some works address the few-
shot learning task. [1] is one representative example. They use a few training
images with ground-truth annotations and build 3D models to bridge the testing
image to those training images with ground-truth. However, in addition to the
requirement of labels, their model could be slow since they run a matching
algorithm during the inference. We would like to build a simple and efficient
model so that we may run the inference with a little overheads. Another work
of weakly supervised part detection is [6]. They detect part landmarks with a
simple prior of the occurrence of object parts which were trained with image-
level labels.

Figure 2: Our model is illustrated in the picture, the image is a 2D image
with an object (bicycle as an example on the left), and the output feature map
of the backbone is semantically meaningful and corresponds to object parts
(bicycle wheel center as examples on the right).

##  3 Proposed Method

The goal of our paper is to detect keypoints or parts of an object on a given
2D image, with a convolutional neural network (CNN). Without training on
ground-truth part labels, we expect the output feature map of the CNN to be
semantically meaningful and able to correspond to parts precisely. As shown by
the example in Figure 2, the input on the left is a 2D bicycle image, and the
output feature maps on the right can capture object parts.

Formally, given an input image noted with $\bm{I}\in\mathbb{R}^{H\times
W\times 3}$, we want to generate a semantically meaningful output feature map
$\bm{O}\in\mathbb{R}^{h\times w\times c_{o}}$ to capture the parts of the
corresponding object where $h$, $w$, and $c_{o}$ correspond to the height,
weight, and number of channels of feature map ($h$ and $w$ can be then up-
sampled to $H$ and $W$, the size of image). The ground-truth part annotations
of the image $\bm{I}$ with $n$ different parts are
$\bm{P}=\\{p_{1},p_{2},...,p_{n}\\}$, and are used for evaluation only.
$p_{i}=[(x_{i1},y_{i1}),...]$ corresponds the locations of the $i$-th part and
may contains more than one item because there may be multiple instances of
some parts (wheels) and each item corresponds to the center of an instance of
the part in the image. To detect the parts, we expect each of the part
$p_{i}$, should be corresponded by at least one channel of $\bm{O}$, whose
values are high around the part center $(x_{ij},y_{ij})$ and are low for the
areas far away from these part instance centers. The rightmost image in Figure
2 shows an example, where $p_{1}$ of the bicycle image is corresponded by the
feature map channel illustrated in the heat-map.

###  3.1 Overall Framework

As Figure 2 shown, given an input image $\bm{I}\in\mathbb{R}^{H\times W\times
3}$, our method first uses an ImageNet-pretrained deep convolutional neural
network (VGG-16[20] for our implementation) to build a backbone feature map
$F\in\mathbb{R}^{h\times w\times c_{i}}$. Then we add an extra convolutional
layer (named part layer) to model and capture parts of the object based on the
backbone feature map. The part layer is initialized by weights generated from
clustering of backbone feature vectors [10]. Then the semantic part
information can be extracted in the output feature
$\bm{O}\in\mathbb{R}^{h\times w\times c_{o}}$, after normalized by another
layer of softmax. The middle part of Figure 2 illustrates the backbone of our
framework, which maps an input image to the output semantic part map
($\bm{O}=\text{PartLayer}\circ\text{Backbone}(\bm{I})$). The inference of this
network is very straightforward and computationally inexpensive. As we will
show in the Figure 3, without further training, the initialized part layer
already learns the semantic parts or key-points reasonably well, with the part
layer initialized by the clustering of feature vectors. However, this
initialization is far from perfect, for example, firing inconsistently on
different images, and firing on different part regions. In the next
subsections, we will discuss the training process of the backbone network,
without ground-truth supervision, to further tune the part layer for a
stronger semantic part discovering.

Figure 3: The initialized part layer predicts reasonable but not perfect part
feature map of bicycle as shown in picture. Each column corresponds to the
same output channel and represents one part, from left to right are: wheel
center, upper part of the wheel, bottom part of the wheel, front part of the
wheel, back part of the wheel Figure 4: This figure shows the training
mechanism, which can be mainly divided into four steps. The first step is to
discover similar images by cosine similarity (on the left of figure). The
second step is to align feature maps based on the spatial transformation
computed via matching (on the upper right). The third step is to generate a
fake ground truth (on the bottom right). The last step is to compute the loss
between aligned feature maps and pseudo ground truth (on the rightmost of
picture).

###  3.2 Training Phase

The intuition of our work is that similar images can be aligned and the
structure of the aligned images should be coherent, thus part detectors should
fire coherently on them. To be specific, given a training image
$\bm{I}_{i}\in\mathbb{R}^{H\times W\times 3}$ without knowing its ground-truth
parts/key-points, we first want to find a small pool of other training images:
$\text{{SimPool}}(\bm{I}_{i})=\\{\bm{I}_{i_{1}},\bm{I}_{i_{2}},\cdots,\bm{I}_{i_{k}}\\}$,
which are visually and semantically similar to $\bm{I}_{i}$. By similar, we
mean they should not have largely different poses, sub-types, illuminations, .
Then the backbone feature maps of all those images are extracted to remove the
low-level details and keep the semantics, noted as $\bm{O}_{i}$ and
$\\{\bm{O}_{i_{1}},\bm{O}_{i_{2}},\cdots,\bm{O}_{i_{k}}\\}$. Before directly
encouraging coherence between these feature maps, we apply some small spatial
transformations:
$\bm{O}^{\prime}_{i_{j}}=\text{{SpatialTrans}}(\bm{O}_{i_{j}})$, in order to
adjust for the small spatial differences between the training image and the
images between the similar pool, and align the feature maps of the similar
images to the canvas of $\bm{O}_{i}$.

Given the aforementioned feature maps $\bm{O}_{i}$ and
$\\{\bm{O}^{\prime}_{i_{1}},\bm{O}^{\prime}_{i_{2}},\cdots,\bm{O}^{\prime}_{i_{k}}\\}$,
supervision for this training image emerges from encouraging coherence between
them, ,

$\text{{Loss}}(\bm{I}_{i})=\text{{Coherence}}(\bm{O}_{i},\bm{O}^{\prime}_{i_{1}},\bm{O}^{\prime}_{i_{2}},\cdots,\bm{O}^{\prime}_{i_{k}})$

The intuition is that if $\bm{O}_{i}$ or $\bm{O}^{\prime}_{ij}$ is largely
different at a specific location of a feature channel compared with the other
features from the similar image pool, we believe it should be a false positive
at that location if the channel is considered semantically meaningful. In the
case the feature channel is not semantically meaningful (the initialized part
layer is not tight or corresponds to background), the coherence can encourage
the feature channel learning something appearing consistently on those images.

Figure 4 shows the training pipeline of our model, and we will discuss each
component in details in the following subsections.

####  3.2.1 Similar image finding

For each training iteration, the first step is to discover a small pool of
images who are similar to the training image, from the full training set. To
determine the similarity between two images $\bm{I}_{a}$ and $\bm{I}_{b}$,
instead of comparing the RGB pixels, we compare the feature maps generating
from the initialized model, , $\bm{O}_{a}$ and $\bm{O}_{b}$. To be specific,
we compare the Jensen-Shannon divergence between the resized feature maps by
$s_{ab}=\text{{Sim}}(\bm{I}_{a},\bm{I}_{b})=\bm{D}_{JS}(\bm{O}_{a},\bm{O}_{b})$.
The pair-wise similarity scores can be pre-computed for the initialized model,
and can also be updated every a few epochs during the training. At the
beginning of each training iteration, we first find the small pool of similar
images by retrieving the top-k images from the training set (or a subset of
the training set in case it is too large), thus:

$\\{\bm{I}_{i_{1}},\cdots,\bm{I}_{i_{k}}\\}=\text{{SimPool}}(\bm{I}_{i})=\\{\bm{I}_{j}:j\in\text{ArgTopK}_{j}(s_{ij})\\}$

Some examples of the similar image pools are visualized in Figure 5. This
process is very fast since the similarity scores are pre-computed and the
similar image pools can also be pre-fetched if necessary.

Figure 5: Examples of similar image pool for bicycle images of the VehiclePart
dataset. The similar images always have similar poses and shapes.

####  3.2.2 Feature map aligning

In order to encourage coherence between the training image and the similar
images, we want to first align them on the feature map level, to adjust for
small variances (small pose/structure/aspect ratio differences). To enhance
weak part responses, we first normalize each feature map with its maximum
value in the spatial dimensions. Given the training image feature map
$\bm{O}_{i}$, we aim at aligning every image feature map $\bm{O}_{ij}$
extracted from the similar image pool to the canvas of $\bm{O}_{i}$, via
RANSAC. Note that $\bm{O}_{i}$ and $\bm{O}_{ij}$ may have different shapes (,
$h_{1}\times w_{1}$ and $h_{2}\times w_{2}$).

To estimate spatial transformation and carry out the alignment, we first
compute the cosine similarity of feature vectors between the aforementioned
feature maps, , $\bm{c}(pq;rs)=\text{{Cos}}(\bm{O}_{i}(pq),\bm{O}_{ij}(rs))$,
where $p\in[1,h_{1}],q\in[1,w_{1}],r\in[1,h_{2}],s\in[1,w_{2}]$. A threshold
is used to find those matched pairs, $M=\\{pqrs:\bm{c}(pq;rs)>th\\}$. We
randomly select a subset from $M$, and estimate the parameters for a spatial
transformation (affine transformation) which maps a coordinate in the similar
image feature map to the original training image feature map
($\hat{pq}=T(rs,\theta)$). Number of in-liner counts are computed by how many
matched in $M$ satisfy the estimated spatial transformation, and the best
transformation with the largest set of in-liner matches are used as the final
spatial transform for the corresponding image pair of the training image and
the similar image, and the aligned feature map can be obtained by transferring
the coordinates on $\bm{O}_{ij}$ to the canvas of $\bm{O}_{i}$. The aligned
feature map of the similar image is:

$\bm{O}^{\prime}_{ij}=\text{{SpatialTrans}}(\bm{O}_{ij})=\text{Warp}(\bm{O}_{ij},T(\cdot,\hat{\theta}))$

####  3.2.3 Pseudo ground-truth generation

The supervision of our training comes from the coherence between the training
image feature map $\bm{O}_{i}$ and the aligned similar image feature maps
$\\{\bm{O}^{\prime}_{ij},j\in[1,k]\\}$. In order to encourage the coherence,
we first generate a pseudo ground-truth for each output channel from the
similar image pool, and then use that to supervise the feature maps as a per-
grid segmentation task. The details of the pseudo ground-truth generation is
illustrated in Algorithm 1. It generates a pseudo ground-truth map of shape
$h\times w\times 1$, by first computing the average feature map of
$\\{\bm{O}^{\prime}_{ij},j\in[1,k]\\}$ and finding the maximum values
iteratively in the averaged tensor. Note that for each channel, we allow at
most 3 spatial positions corresponding to that, to encourage the
discriminativeness. The generated pseudo ground-truth map indicates the
coherent firing pattern of the image pool with images similar to the training
image.

Algorithm 1 generate pseudo GT

1:$\\{\bm{O}^{\prime}_{ij},j\in[1,k]\\}$: the group of aligned feature maps,
with shape of $h\times w\times c$ for each of them.

2:function generate($\\{\bm{O}^{\prime}_{ij},j\in[1,k]\\}$)

3: $\bar{O}\leftarrow\text{Mean}(\\{\bm{O}^{\prime}_{ij},j\in[1,k]\\})$

4: $h,w,c\leftarrow\bar{O}.shape$

5: $\bm{GT}_{pseudo}\leftarrow-ones(h,w)$

6: $cnt\leftarrow zeros(c)$

7: for $i=0\to h*w$ do

8: $h_{i},w_{i},c_{i}=\bar{O}.argmax()$

9: $\bm{GT}_{pseudo}[h_{i},w_{i}]\leftarrow c_{i}$

10: $\bar{O}[h_{i},w_{i},:]\leftarrow-1$

11: $cnt[c_{i}]\leftarrow cnt[c_{i}]+1$

12: if $cnt[c_{i}]==3$ and $c_{i}<c-1$ then

13: $\bar{O}[:,:,c_{i}]\leftarrow-1$

14: end if

15: end for

16: return $\bm{GT}_{pseudo}$

17:end function

####  3.2.4 Loss function and supervision

With our intuition, the detection training can be supervised by the pseudo
ground-truth on the output feature maps. To encourage the aligned feature maps
from different images to be coherent, we simply request all of them to be
similar to the pseudo ground-truth. To this end, we adopt a per-pixel
classification on the averaged aligned feature maps generated from
$\bar{\bm{O}}=\text{Average}(\\{\bm{O}_{i},\bm{O}^{\prime}_{i1},\cdots,\bm{O}^{\prime}_{ik}\\})$.
The loss is computed with log likelihood loss $\bm{L}_{nll}$, :

$\bm{L}=\bm{L}_{nll}(\log(\bm{\overline{O}}^{i}),\bm{GT}_{pseudo})$

###  3.3 Testing Phase

The test phase is very simple and fast. After the images going through the
backbone and normalization, we have feature maps. Then, we generate anchors
from feature maps and then select some of them by Non-Maximum Suppression as
our prediction. About the evaluation and quantitative results, we will talk
about it in Section 4.4.

##  4 Experiments

In this section, we conduct comprehensive experiments on several datasets with
landmarks/keypoints for multiple domains such as vehicles, birds and human
faces. Our model outperforms several baselines and state-of-the-art
alternatives. We carry out several ablation experiments that demonstrate the
effect of the individual components in our model. Finally, we analyze
qualitative examples which demonstrate that our method is able to discover
semantically meaningful part representations in an unsupervised manner.

###  4.1 Implementation Details

We implement our method with PyTorch [16], and we train our models with a
single nVIDIA GPU. We use a VGG16 network [20] as our backbone, and initialize
it with ImageNet [19] pre-trained weights. A part learning layer is added
after the VGG16 to discover and detect object parts. This layer is initialized
with a part dictionary obtained by running an unsupervised clustering
algorithm on VGG16 feature vectors on the training dataset [10]. From the
illustration in Figure 3, we can observe that these clusters correspond
reasonably well to object parts with some errors, which indicates the quality
of this strong baseline. As a classifier, we apply a softmax layer on top of
the channel-normalized feature vectors.

###  4.2 Datasets

VehiclePart. The VehiclePart [26] dataset is an extension of Pascal 3D+ [28]
and contains images of vehicles including aeroplanes, bicycles, buses, cars,
motorbikes and trains. Each category contains about 300 to 2000 training
images and similar amounts of testing images. In addition to the landmark
labels provided in Pascal 3D+, the VehiclePart dataset additionally defines
more than 10 semantic parts for each vehicle type.

CUB The CUB [25] dataset contains 10,000 bird images with 200 categories of
birds with various bird poses and from different camera views. The training
set holds 5,994 images while the test set holds 5,794 images. Each image has
15 part landmarks.

CelebA. The CelebA [12] dataset contains about 200k human face images.
Following the settings used in [32], we use the CelebA train set excluded MAFL
[33] (subset of CelebA) test set as our train set (161962 images in total),

and use the MAFL test set (1000 images in total) as our test set. The dataset
provides two different settings, one is unaligned images with annotated
bounding box around the face. The other setting involvescroped and aligned
faces. We apply our method to both setups. Each image is annotated with 5
facial landmarks.

MAFL. MAFL [33] is a subset of CelebA. The only difference to CelebA is the
size of the training set (19k in MAFL, 162k in CelebA).

###  4.3 Training/Testing Details

During both training and testing, the short side of the input image is resized
to 224 pixels so that all objects are under roughly the same scales. The VGG16
backbone is fixed and only the part discovery layer is trained. The part
discovery layer generates 513 channels, including 512 part clusters
initialized from cluster centers as discussed in [10] and one background
channel. For the training phase, we use the Adagrad optimizer with a learning
rate of $5\times 10^{-3}$ and a learning decay of 0.1. The top-k used for
building similar image pools is 15, inside a subset of training set with 2000
images. This reduces the memory used for storing the pair-wise similarity
scores as discussed in Subsection 3.2.1. The cosine similarity threshold for
feature map alignment is 0.6. We use an affine transformation to warp feature
maps between similar images. For the testing phase, the IOU threshold of the
non-maximum suppression is 0.3 during the post-processing.

###  4.4 Quantitative Results

In this section, we discuss quantitative results and comparisons, and
illustrate that our novel idea is promising and our innovative designed method
is more effective than other baselines.

First, following the designs in the VehiclePart dataset, we use mean average
precision (mAP) with an IOU threshold of 0.5 as our evaluation on the part
bounding boxes provided in the VehiclePart. Moreover, we use mAP with an L2
distance threshold of 0.1 as the performance metric for both the CUB and
CelebA dataset. In this evaluation, our method will be compared with
clustering based part discovery [10] and InterpretableCNN [31]. The clustering
based part discovery performs unsupervised clustering on the feature vectors
from a backbone CNN, and uses these cluster centers as part detectors.
Regarding the InterpretableCNN, it adds a regularizer on a traditional CNN, by
assigning spatial masks to each channel of the CNN and using the corresponding
mask to supervise and regularize the CNN. Thus those channels can better
correspond to parts. We call them Clustering and MaskRegularized in the
remainder of this paper.

In addition to the aforementioned baselines, we also compare our results on
the CelebA and CUB dataset with recent unsupervised methods [7, 6, 24, 32] and
other supervised methods [29, 35, 21, 34]. In these comparisons, we follow
their evaluation protocol and use L2 distance normalized by either the pupil
distance (for CelebA) or the object bounding box size (for CUB). In the
experiments, we follow the common mechanism of using a linear regressor
trained on part of the training set to map the unsupervised predictions to the
manually labeled landmarks.

mAP |  Aero |  Bicycle |  Bus |  Car |  Mbike |  Train |  Average
---|---|---|---|---|---|---|---
Clustering |  $16.7$ |  $54.6$ |  $28.7$ |  $32.2$ |  $32.2$ |  $17.6$ |  $29.7$
MaskReg |  $\mathbf{18.7}$ |  $56.9$ |  $37.4$ |  $29.4$ |  $37.9$ |  $\mathbf{21.0}$ |  $33.6$
Ours |  $\mathbf{18.7}$ |  $\mathbf{66.7}$ |  $\mathbf{46.6}$ |  $\mathbf{35.5}$ |  $\mathbf{39.6}$ |  19.4 |  $\mathbf{37.8}$
Table 1: Results of unsupervised part detection methods on VehiclePart,
including aeroplane, bicycle, bus, bus, motorbike, and train. Our model can
predict bounding boxes more accurately than Clustering and MaskRegularized
methods (see details in the second paragraph of Section 4.4).

####  4.4.1 VehiclePart

In the Table 1, we show the mean AP with IOU threshold of 0.5 on the
VehiclePart dataset. Comparing with the mAP of $29.7$ for the Clustering
method and the mAP of $33.6$ for the MaskRegularized method, our average
performance of the six categories is $37.8$ mAP, which achieves an improvement
of at least 12.5%. The results illustrate that our method is promising
compared with other unsupervised methods. Different from the Clustering
method, we take both local appearance of the part in one image and inter-image
part alignment into consideration when learning the part dictionary (the part
layer weights). The MaskRegularized method mainly uses single peak feature
maps to regularize the CNN channels, and the supervision is not very strong.
For example, during the training, a bad channel feature map may lead to a bad
mask. In addition, it only allows for a single peak per channel for each
training example, while our method allows for multiple peaks (up to three) in
one feature map, which can help for discovering parts such as the wheels of a
car.

mAP |  Unalign |  Align
---|---|---
Clustering |  $71.8$ |  $92.6$
Ours |  $\mathbf{90.9}$ |  $\mathbf{97.9}$
Table 2: Compared with our baseline on CelebA aligned and unaligned set (both 200k images, including 5 parts), our result is 19.1 or 5.3 points ahead. Error |  Unalign |  Align
---|---|---
Clustering[27] |  $13.8$ |  $6.0$
[7] |  $15.0$ |  $-$
[6] |  $8.4$ |  $-$
[24] |  $-$ |  $5.33$
[32] |  $-$ |  $\mathbf{3.15}$
Ours |  $\mathbf{6.3}$ |  $5.0$
Table 3: Results of applying unsupervised landmarks detection methods on CelebA aligned and unaligned set (both 200k images, including 5 parts). Our result achieves state-of-the-art on unaligned set and is close to [32] on align set. Error |  MAFL align
---|---
Fully supervised |  CFAN[29] |  $15.84$
|  TCDCN[35] |  $7.59$
|  Cascaded CNN[21] |  $9.73$
|  MTCNN[34] |  $\mathbf{5.39}$
Unsupervised |  [24] |  $6.67$
|  [22] |  $5.8$
|  [32] |  $\mathbf{3.15}$
|  Ours |  $4.9$
Table 4: Compared with both unsupervised and supervised part detection methods
on MAFL aligned set, our result surpasses all supervised methods and only
worse than [32] of unsupervised methods.

####  4.4.2 CelebA human face dataset

To further verify the effectiveness of our method and its applicability to
other datasets, we test our method on CelebA, which has a training set of 80
times larger compared with the VehiclePart dataset. In Table 2, we report the
mean AP based on the L2 distance threshold. Compared with the Clustering
method, our method improves $19.1\%$ on the unaligned setting and $5.3\%$ on
the aligned setting. Hence, we observe that our method a similar effect as on
the VehiclePart dataset, namley that our method improves the part layer
weights.

In addition, we also compare our method with the state-of-the-art unsupervised
methods. In the Table 3, we evaluate results by L2 distance normalized by
pupil distance after linear regression and obtain the error of $6.3\%$ for
unaligned and $5.0\%$ for aligned images. Notably, we achieve the state-of-
the-art performance in the more challenging unaligned setting. [32] is
slightly better ($1.8\%$ of pupil distance) compared with our method in the
aligned setting.

We surpass all supervised methods on the MAFL dataset in the aligned setting
(with an error of $4.9\%$), while being slightly worse than [32] when
comparing with unsupervised methods as shown in Figure 4.

mAP |  Bird
---|---
Clustering |  $26.9$
Ours |  $\mathbf{28.2}$
Table 5: Compared with baseline on CUB 10k images of 200 categories including 15 parts, our result is 1.3 points better. Error |  Bird
---|---
Clustering |  $13.9$
[6] |  $11.5$
Ours |  $\mathbf{11.1}$
Table 6: Results of applying unsupervised landmarks detection methods on CUB. Our result reaches the best results among them. mAP |  Configuration |  Aeroplane |  Bicycle |  Bus |  Car |  Motorbike |  Train |  Average
---|---|---|---|---|---|---|---|---
Top-k |  5 |  $16.4$ |  $64.6$ |  $41.9$ |  $32.6$ |  $\mathbf{39.9}$ |  $14.4$ |  $35.0$
|  10 |  $16$ |  $66.2$ |  $46.5$ |  $\mathbf{37.3}$ |  $38.8$ |  $\mathbf{20.9}$ |  $37.6$
|  15 |  $\mathbf{18.7}$ |  $\mathbf{66.7}$ |  $46.6$ |  $35.5$ |  $39.6$ |  $19.4$ |  $\mathbf{37.8}$
|  20 |  $18.5$ |  $64.1$ |  $\mathbf{48.3}$ |  $36.5$ |  $36.7$ |  $19.2$ |  $37.2$
Spatial transform |  affine |  $16$ |  $66.2$ |  $\mathbf{46.5}$ |  $\mathbf{37.3}$ |  $38.8$ |  $\mathbf{20.9}$ |  $\mathbf{37.6}$
|  translation |  $\mathbf{18.7}$ |  $\mathbf{68.6}$ |  $44.5$ |  $36.5$ |  $\mathbf{40.3}$ |  $16.6$ |  $37.5$
|  homography |  $16.9$ |  $65.3$ |  $45.6$ |  $35.7$ |  $40.2$ |  $13.1$ |  $36.1$
|  without transform |  $15.3$ |  $64.4$ |  $42.2$ |  $33.3$ |  $36.4$ |  $16.5$ |  $34.7$
Suppress neighbour |  0 |  $\mathbf{16}$ |  $66.2$ |  $46.5$ |  $\mathbf{37.3}$ |  $38.8$ |  $\mathbf{20.9}$ |  $37.6$
|  1 |  $15.6$ |  $69.2$ |  $\mathbf{47.3}$ |  $34.3$ |  $\mathbf{41.9}$ |  $17.1$ |  $37.6$
|  2 |  $15.9$ |  $\mathbf{73.4}$ |  $45.7$ |  $\mathbf{37.3}$ |  $41.6$ |  $15.3$ |  $\mathbf{38.2}$
Table 7: This figure shows three ablation studies about different unit in our
pipeline. The detailed analysis is discuss in Section 4.5

####  4.4.3 CUB dataset

The objects of VehiclePart and CelebA datasets are considered rigid objects (,
vehicles and faces), so we would like to verify our method’s ability to
discovery non-rigid objects parts/landmarks, such as deformable birds. The CUB
dataset is challenging because it consists of 200 bird classes in various
poses (see examples in Figure 9), , standing, swimming, or flying, as well as
the different camera viewpoints. Our performance is $28.2\%$ for mAP as shown
in the Table 5 while the Clustering method achieves $26.9\%$. Notably, the [6]
method trains with the image-level label while our method is free of using
these extra labels. Despite using less supervision, we still achieve state-of-
the-art performance as shown in Table 6.

Figure 6: Visualization of the bicycle (row 1-3) and the motorbike (row 4-6).
We show the corresponding feature map for the first five parts: wheel center,
upper part of the wheel, bottom part of the wheel, front part of the wheel,
back part of the wheel. Figure 7: Visualization of the bus. We show the
corresponding feature map for the first five parts: wheel, side window and
side body, headlight, license plate, display screen. Figure 8: Visualization
of the human face (row 1 is unaligned and row 2 is aligned). We show all five
parts feature map: left eye, right eye, nose, left mouth, right mouth Figure
9: CUB dataset contains birds with different shapes and colorful feathers,
various bird pose, multiple camera viewpoints. We take the beak part as an
example. Our method can discover parts clearly and robustly.

###  4.5 Ablation Studies

To explore the individual contribution of each component in our pipeline, we
conduct a detailed ablation study and illustrate the performance in Table 7.
To be more specific, we mainly investigate three different components in our
method and will discuss them in the following paragraphs.

Firstly, top-k in similar images finding, which is the number of the most
similar images compared with the given training image, is important to our
training. The performance will drop by more than 2 points when we use a
smaller similar image pool with only 5 images. When the top-k is too big (goes
up to 20), the performance may drop a little bit, in the case that the
training sets for some categories are not large enough to provide a large
amount of similar images.

Second one is a transformation in the feature map alignment unit. As the
results shown, our method has a good robustness to different transformations.
Even using simple translation, we can achieve promising results. On the other
hand, without transformation to align feature maps, the performance drops by
about 3 points. The conclusion is that our idea about alignment is necessary
to provide solid supervision.

We also diagnose the process of pseudo ground-truth generation. In order to
comply with the dataset that one part may have more than one instance (, a car
usually has more than one wheel), we allow multiple locations for a single
channel when building the pseudo ground-truth. To avoid selecting redundant
nearby locations, we suppress the average feature map values for the selected
channel around the selected location. The radius to be suppressed are 0, 1 and
2, and we report the performance for these settings in the third block of
Table 7. We can see the results are significantly improved for bicycle and
motorbike with the suppression. However, other categories may have some
regressions. So we choose to use a suppression radius of 0 in our final
settings.

error |  unalign |  align
---|---|---
CelebA |  $6.1$ |  $\mathbf{5.0}$
MAFL |  $\mathbf{7.1}$ |  $4.9$
Table 8: We research about the influence of using different size of training
set to our method. The results illustrate that our method is not sensitive to
quantity

Finally, to verify our method robust on small training size, we train our
model and fit the linear regressor both on MAFL train set and test on MAFL
test set. The performances in Table 8 show that our method is stable when the
number of training images decreases.

###  4.6 Qualitative Results

As shown from Figure 6 to Figure 9, our method provides feature maps that have
strong relation with parts, even those with multiple instances. If we look
closely at the Figure 7, the $5th$ part (last row), display screen located on
the upper part of the bus head, can be detected in the right location even
without any real display screen there as the first and third bus shown. That
prediction proves that our method based on alignment is effective and it can
predict part location exactly only based on other images’ information.
Besides, Figure 9 verifies that our method is robust to object shape, posture,
and viewpoints.

##  5 Conclusion

In this paper, we investigate the problem of unsupervised parts or landmarks
discovering, which is a very important topic to understand images better
without extra expensive and time-consuming annotations. Based on the intuition
that the detected parts should fire in a similar manner on different images,
we propose a novel method which first finds and aligns a set of similar
images, and then encourages the coherence between the feature maps from those
images, to make them to learn tighter and more consistent parts. The proposed
method is very simple and efficient during the inference phase, which is
simply a feed-forward of a CNN. Extensive experiments have been carried out on
several different datasets and domains, and our method is consistently better
than our baselines and other alternatives. Further works including
unsupervisedly parsing objects or using the parts to improve
classification/occlusion could be done based on our work, and we remain them
as the future works.

## References

  * [1] Yutong Bai, Qing Liu, Lingxi Xie, Weichao Qiu, Yan Zheng, and Alan L Yuille.  Semantic part detection via matching: Learning to generalize to novel viewpoints from limited training data.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7535–7545, 2019.
  * [2] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1302–1310. IEEE Computer Society, 2017.
  * [3] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7291–7299, 2017.
  * [4] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun.  R-FCN: object detection via region-based fully convolutional networks.  In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 379–387, 2016.
  * [5] Jan Hendrik Hosang, Rodrigo Benenson, and Bernt Schiele.  Learning non-maximum suppression.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6469–6477. IEEE Computer Society, 2017.
  * [6] Zixuan Huang and Yin Li.  Interpretable and accurate fine-grained recognition via region grouping.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8662–8672, 2020.
  * [7] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang, and Jan Kautz.  Scops: Self-supervised co-part segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 869–878, 2019.
  * [8] Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang.  Acquisition of localization confidence for accurate object detection.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notes in Computer Science, pages 816–832. Springer, 2018.
  * [9] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 371–386, 2018.
  * [10] Adam Kortylewski, Qing Liu, Huiyu Wang, Zhishuai Zhang, and Alan Yuille.  Combining compositional models and deep networks for robust object classification under occlusion.  In The IEEE Winter Conference on Applications of Computer Vision, pages 1333–1341, 2020.
  * [11] Yann Lecun, Yoshua Bengio, and Geoffrey E Hinton.  Deep learning.  Nature, 521(7553):436–444, 2015.
  * [12] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 3730–3738. IEEE Computer Society, 2015.
  * [13] Jonathan L Long, Ning Zhang, and Trevor Darrell.  Do convnets learn correspondence?  In Advances in neural information processing systems, pages 1601–1609, 2014.
  * [14] Ross Messing, Chris Pal, and Henry Kautz.  Activity recognition using the velocity histories of tracked keypoints.  In 2009 IEEE 12th international conference on computer vision, pages 104–111. IEEE, 2009.
  * [15] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In European conference on computer vision, pages 483–499. Springer, 2016.
  * [16] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in neural information processing systems, pages 8026–8037, 2019.
  * [17] Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G Derpanis, and Kostas Daniilidis.  6-dof object pose from semantic keypoints.  In 2017 IEEE international conference on robotics and automation (ICRA), pages 2011–2018. IEEE, 2017.
  * [18] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 91–99, 2015.
  * [19] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li.  Imagenet large scale visual recognition challenge.  Int. J. Comput. Vis., 115(3):211–252, 2015.
  * [20] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [21] Yi Sun, Xiaogang Wang, and Xiaoou Tang.  Deep convolutional network cascade for facial point detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3476–3483, 2013.
  * [22] James Thewlis, Hakan Bilen, and Andrea Vedaldi.  Unsupervised learning of object frames by dense equivariant image labelling.  In Advances in neural information processing systems, pages 844–855, 2017.
  * [23] James Thewlis, Hakan Bilen, and Andrea Vedaldi.  Unsupervised learning of object landmarks by factorized spatial embeddings.  In Proceedings of the IEEE international conference on computer vision, pages 5916–5925, 2017.
  * [24] James Thewlis, Hakan Bilen, and Andrea Vedaldi.  Unsupervised learning of object landmarks by factorized spatial embeddings.  In Proceedings of the IEEE international conference on computer vision, pages 5916–5925, 2017.
  * [25] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [26] Jianyu Wang, Cihang Xie, Zhishuai Zhang, Jun Zhu, Lingxi Xie, and Alan Yuille.  Detecting semantic parts on partially occluded objects.  arXiv preprint arXiv:1707.07819, 2017.
  * [27] Jianyu Wang, Zhishuai Zhang, Cihang Xie, Yuyin Zhou, Vittal Premachandran, Jun Zhu, Lingxi Xie, and Alan Yuille.  Visual concepts and compositional voting.  arXiv preprint arXiv:1711.04451, 2017.
  * [28] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese.  Beyond pascal: A benchmark for 3d object detection in the wild.  In IEEE winter conference on applications of computer vision, pages 75–82. IEEE, 2014.
  * [29] Jie Zhang, Shiguang Shan, Meina Kan, and Xilin Chen.  Coarse-to-fine auto-encoder networks (cfan) for real-time face alignment.  In European conference on computer vision, pages 1–16. Springer, 2014.
  * [30] Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell.  Part-based r-cnns for fine-grained category detection.  In European conference on computer vision, pages 834–849. Springer, 2014.
  * [31] Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu.  Interpretable convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8827–8836, 2018.
  * [32] Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee.  Unsupervised discovery of object landmarks as structural representations.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2694–2703. IEEE Computer Society, 2018.
  * [33] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Facial landmark detection by deep multi-task learning.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI, volume 8694 of Lecture Notes in Computer Science, pages 94–108. Springer, 2014.
  * [34] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Facial landmark detection by deep multi-task learning.  In European conference on computer vision, pages 94–108. Springer, 2014.
  * [35] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Learning deep representation for face alignment with auxiliary attributes.  IEEE transactions on pattern analysis and machine intelligence, 38(5):918–930, 2015.
  * [36] Zhishuai Zhang, Cihang Xie, Jianyu Wang, Lingxi Xie, and Alan L. Yuille.  Deepvoting: A robust and explainable deep network for semantic part detection under partial occlusion.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 1372–1380. IEEE Computer Society, 2018.
  * [37] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Object detectors emerge in deep scene cnns.  arXiv preprint arXiv:1412.6856, 2014.


# Unsupervised Multiplex Graph Learning
with Complementary and Consistent Information

Liang Peng  School of Computer Science and Engineering, University of
Electronic Science and Technology of China, Chengdu, China ,  Xin Wang  School
of Computer Science and Engineering, University of Electronic Science and
Technology of China, Chengdu, China and  Xiaofeng Zhu  School of Computer
Science and Engineering, University of Electronic Science and Technology of
China, Chengdu, China
Shenzhen Institute for Advanced Study, University of Electronic Science and
Technology of China, Shenzhen, China

(2023)

###### Abstract.

Unsupervised multiplex graph learning (UMGL) has been shown to achieve
significant effectiveness for different downstream tasks by exploring both
complementary information and consistent information among multiple graphs.
However, previous methods usually overlook the issues in practical
applications, i.e., the out-of-sample issue and the noise issue. To address
the above issues, in this paper, we propose an effective and efficient UMGL
method to explore both complementary and consistent information. To do this,
our method employs multiple MLP encoders rather than graph convolutional
network (GCN) to conduct representation learning with two constraints, i.e.,
preserving the local graph structure among nodes to handle the out-of-sample
issue, and maximizing the correlation of multiple node representations to
handle the noise issue. Comprehensive experiments demonstrate that our
proposed method achieves superior effectiveness and efficiency over the
comparison methods and effectively tackles those two issues. Code is available
at https://github.com/LarryUESTC/CoCoMG.

Multiplex graph learning, Graph representation learning, Unsupervised
learning.

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
31st ACM International Conference on Multimedia; October 29-November 3, 2023;
Ottawa, ON, Canada††booktitle: Proceedings of the 31st ACM International
Conference on Multimedia (MM ’23), October 29-November 3, 2023, Ottawa, ON,
Canada††price: 15.00††doi: 10.1145/3581783.3611971††isbn:
979-8-4007-0108-5/23/10††ccs: Computing methodologies Machine learning††ccs:
Computing methodologies Unsupervised learning

##  1\. Introduction

Multiplex graph is a type of data that consists of multiple graphs, where each
graph represents a different type of relationship among nodes (Battiston et
al., 2017; Gan et al., 2022; Liu et al., 2022). This kind of data is prevalent
in various practical applications, such as social networks and biological
networks (Zhang et al., 2019; Zhu et al., 2022a; Yuan et al., 2021; Li et al.,
2021). Recently, there has been growing interest in unsupervised multiplex
graph learning (UMGL) due to its ability to extract valuable information from
multiple relationships among nodes without relying on label information (Park
et al., 2020; Chu et al., 2019; Zhang et al., 2022; Wang et al., 2022b).

The success of existing UMGL methods lies in that they are able to explore
both complementary information and consistent information among multiple
graphs, so that UMGL can provide more comprehensive information than graph
learning on the single graph (Chu et al., 2019; Zhou et al., 2020; Xie et al.,
2023). The complementary information refers to different types of
relationships that can supply each other and provide a more comprehensive
understanding of the hidden patterns or structures in the data (Cen et al.,
2019). For example, friendships and communication relationships provide
complementary information to identify individuals belonging to the same social
circle (Wang et al., 2022a). The consistent information means consistent
relationships among nodes or structures across multiple graphs, which can help
the model recognize the same category effectively. For example, the
consistency of gene expression and protein-protein interactions is helpful to
identify categories in biological networks (Zhu et al., 2022b; Li et al.,
2021). Obviously, either complementary information or consistent information
often plays an important role in UMGL.

Various methods have been proposed in the literature (Park et al., 2020; Wang
et al., 2021; Jing et al., 2021; Zhu et al., 2022b) to explore and leverage
either complementary information or consistent information. For instance, Jing
et al. apply graph convolutional networks (GCN) (Kipf and Welling, 2016) to
first obtain representations from each graph and then to fuse these
representations by an attention mechanism (Vaswani et al., 2017), aiming at
exploring complementary information among multiple graphs (Jing et al., 2021).
However, its effectiveness is limited by the noisy information as it ignores
the intrinsic correlation (i.e., consistent information) across different
graphs. To address this limitation, recent researchers (Park et al., 2020;
Wang et al., 2021; Zhu et al., 2022b; Jing et al., 2022a) have explored
contrastive learning techniques (Oord et al., 2018) to simultaneously explore
complementary and consistent information for UMGL. For example, to explore the
consistent information, Park  et al. conduct contrastive learning between node
representations and graph representations (Park et al., 2020; Jing et al.,
2021) and Zhu  et al. conduct contrastive learning among node representations
from different graphs (Zhu et al., 2022b). Thus, it is crucial to learn both
complementary information and consistent information for conducting UMGL.

Figure 1. The flowchart of the proposed CoCoMG. Given the original node
feature matrix $\mathbf{X}$, CoCoMG employs MLP encoders with two constraints
($\mathcal{L}_{LP}$ and $\mathcal{L}_{CCA}$) to generate representation
$\mathbf{Z}^{v}$ ($v=1,...,V$). Specifically, the local preserve loss
$\mathcal{L}_{LP}$ is designed to extract the complementary information of
every graph and deal with the out-of-sample issue. The canonical correlation
analysis (CCA) loss $\mathcal{L}_{CCA}$ is designed to extract the consistent
information among all graphs and deal with the noise issue. Finally, the final
representation $\mathbf{Z}$ is obtained for downstream tasks by averagely
fusing all representations.

Despite the important advances, previous UMGL methods for simultaneously
considering complementary and consistent information usually overlook some
issues, such as the out-of-sample issue (Ma et al., 2018) and the noise issue
(Zhao et al., 2017), resulting in a gap between the existing methods and their
practical applications. On the one hand, previous UMGL methods (Park et al.,
2020; Wang et al., 2021; Jing et al., 2021; Zhu et al., 2022b; Han et al.,
2023) can not directly infer representations for unseen nodes because they do
not directly generate a prediction model to predict unseen nodes.
Specifically, the key issue for either the prediction model generation or the
UMGL is the message-passing mechanism, which needs to aggregate information
from nodes. In real applications, aggregating information may be efficient for
the single graph, but it is usually inefficient for multiple graphs
(Veličković, 2022). On the other hand, previous methods cannot effectively and
efficiently deal with noise for UMGL. For example, previous UMGL methods (Zhu
et al., 2021; Park et al., 2020; Wang et al., 2021; Zhu et al., 2022b; Zhang
et al., 2023) investigate contrastive learning techniques to explore
consistent information, but noise (e.g., incorrect connections) may be
aggregated into the process of representation learning, resulting in
ineffective UMGL. Additionally, many previous methods require constructing and
embedding positive and negative pairs for every graph, which is
computationally inefficient. Overall, it is challenging to effectively and
efficiently explore both complementary and consistent information for UMGL.

To overcome the above issues, in this paper, we explore complementary and
consistent information in a unified framework for UMGL by proposing an
effective and efficient UMGL method CoCoMG, i.e., learning Complementary and
Consistent information for Multiplex Graph in practical scenarios, as shown in
Figure 1. Firstly, we explore the complementary information for UMGL by
applying the Multi-Layer Perceptron (MLP) encoders and designing a local
preserve objective function to achieve effectiveness and efficiency, as well
as to tackle the out-of-sample issue. Specifically, the representations
generated by the MLP encoders are with low computational cost (i.e.,
efficiency), and the representations are expressive enough to characterize the
graph structure (i.e., effectiveness). Moreover, the MLP encoders implicitly
learn the mappings from the features to the graph structure, enabling to
directly predict the embedding of unseen nodes only with their features.
Secondly, we explore the consistent information by conducting an objective
function that maximizes the correlation of multiple representations of nodes
(i.e., effectiveness) without constructing negative pairs (i.e., efficiency).
Meanwhile, the consistent information extraction can also tackle the noise
issue as it can balance the local preserve objective function. As a result,
CoCoMG can effectively and efficiently explore both complementary and
consistent information to deal with the out-of-sample issue and the noise
issue.

Compared to previous UMGL methods, the main contributions of our method can be
summarized as follows:

  * •

The proposed method can tackle the out-of-sample issue and the noise issue for
UMGL. This could reduce the gap between the UMGL methods and their practical
applications.

  * •

The proposed method achieves effectiveness and efficiency for exploring both
complementary and consistent information for UMGL. Comprehensive experiments
verified them by comparing CoCoMG with comparison methods on real datasets.

##  2\. Methodology

Notations. Let $\left\\{\mathbf{A}^{v}\in\mathbb{R}^{N\times
N}\right\\}^{V}_{v=1}$ be a set of graphs, where $N$ is the number of node
samples and $V$ is the number of graphs. $\mathbf{X}\in\mathbb{R}^{N\times D}$
denotes original node features, where $D$ is the dimension of node features.
The goal of the UMGL methods is to learn low-dimensional representations
$\mathbf{Z}\in\mathbb{R}^{N\times d}$ for various downstream tasks, where $d$
is the dimension of the learned representations, and $d\ll D$.

###  2.1. Motivation

Although previous UMGL methods have demonstrated the importance of extracting
complementary and consistent information, a few of them have considered the
issues in practical scenarios, such as the out-of-sample issue and the noise
issue.

First, the existing UMGL methods (Park et al., 2020; Jing et al., 2021; Peng
et al., 2022; Pan and Kang, 2021) do not generate a specific representation
model to directly infer representations for unseen nodes, which is known as
the out-of-sample issue. Specifically, previous UMGL methods heavily relied on
the message-passing mechanism encoder (e.g., GCN) to obtain node
representations, which use multiple graphs to aggregate hidden representations
from neighbors in each layer. However, such aggregation from seen nodes to
unseen nodes requires reconstructing multiple graphs, which is time-consuming
and complex in practical applications. Although some methods (Hamilton et al.,
2017; Velickovic et al., 2018; Ma et al., 2018) have touched this issue by
avoiding reconstructing the graph within the single graph, they still rely on
neighbor information aggregation for inferring the representations of unseen
nodes, which leads to inefficiency in UMGL.

Second, in practical applications, the graph structure usually contains noisy
edges, which can be even more severe in UMGL with multiple graphs. This
results in the noise issue. Although previous methods (Park et al., 2020; Jing
et al., 2021; Wang et al., 2021) have considered using contrastive learning to
explore consistent information to resist noisy information, their
effectiveness might significantly degenerate for dealing with noisy graphs
(verified in Section 3.4). The main reason is possible that contrastive
learning involves aggregating representations of positive pairs and separating
representations of negative pairs. This process necessitates designing and
constructing suitable pairs of positive and negative pairs to achieve
effectiveness. However, whether it is searching for suitable positive and
negative pairs or representing them, both can be impacted by noisy
information, resulting in ineffectiveness in practical applications. Moreover,
obtaining the representations of positive and negative pairs can also be
inefficient in multiple graph learning.

To address the aforementioned issues, we propose an efficient and effective
UMGL method (i.e., CoCoMG) with the framework presented in Figure 1. CoCoMG
extracts the complementary information for every graph (i.e., Section 2.2) and
the consistent information among all graphs (i.e., Section 2.3).

###  2.2. Complementary information extraction

The above analysis implies that previous methods predominantly rely on the
message-passing mechanism to explore complementary information, without
adequately considering their applicability to UMGL. To address this
limitation, one of the straightforward methods is to use the MLP encoder
instead of the message-passing mechanism. However, the MLP encoder is hard to
capture local structures and relationships among nodes, easily leading to
inferior performance.

In this paper, we introduce an efficient and effective method to explore the
complementary information for every graph based on the following three
conditions: (i) Deep. Deep learning models aim at stacking multiple layers to
gradually extract complex and high-level features (LeCun et al., 2015); (ii)
High-order proximity. Multi-hop relationships among nodes have been
demonstrated to provide comprehensive relationships among nodes (Zhu et al.,
2018; Jing et al., 2021); (iii) Smooth. Either traditional graph learning
methods (e.g., shallow learning) or deep graph learning methods (e.g., GCN)
have been demonstrated to preserve the local structure among nodes for
achieving effectiveness (Dong et al., 2016; Kipf and Welling, 2017).

Deep. Given $\mathbf{X}$, we first employ the MLP to extract the
representation for nodes in every graph as follows:

(1) $\mathbf{Z}^{v}=\mathcal{F}_{\Theta^{v}}^{v}(\mathbf{X}),$

where $\mathbf{Z}^{v}$ is the representations matrix of nodes in the $v$-th
graph obtained by the MLP encoder $\mathcal{F}_{\Theta^{v}}^{v}$ with the
parameters $\Theta^{v}$. The parameters of the MLP encoders are not shared
across different graphs as every MLP encoder has unique sets of parameters
that are optimized for the specific graph.

High-order proximity. We define an adjacency matrix to capture the
relationship between nodes and their multi-hop neighbors (e.g., two-hop and
three-hop neighbors), denoted by
$\mathbf{W}^{v}=\mathbf{A}^{v}(\mathbf{A}^{v})^{T}$ with two-hop neighbors,
which results in a more informative representation of nodes. In contrast, the
original adjacency matrix $\mathbf{A}^{v}$ represents the relationship (e.g.,
similarity) between nodes and their one-hop neighbors, and thus is usually
sparse and often lacks information.

Smooth. We encourage neighbor nodes close as much as possible in the embedding
space, and CoCoMG designs the following objective function to preserve the
local structure:

(2)
$\mathcal{L}_{LP}=-\sum_{v=1}^{V}\sum_{i=1}^{N}log\left(\mathbf{w}_{ij}^{v}\overline{\mathcal{S}}(\mathbf{z}_{i}^{v},\mathbf{z}_{j}^{v})\right),$

where $\mathbf{w}^{v}_{ij}\in\mathbf{W}^{v}$ is the strength or similarity
between the $i$-th node and its multi-hop neighbor,
$\overline{\mathcal{S}}(\mathbf{z}_{i}^{v},\mathbf{z}_{j}^{v})$ is the
normalized value of the similarity score
$\mathcal{S}(\mathbf{z}_{i}^{v},\mathbf{z}_{j}^{v}):=exp({cos(\mathbf{z}_{i}^{v},\mathbf{z}_{j}^{v})})$,
and $exp()$ is involved to scale the value of the cosine similarity.

Tackling the out-of-sample issue. To infer the representation for unseen
nodes, previous UMGL methods need to reconstruct multiple graphs (Jing et al.,
2021; Wang et al., 2021, 2022c; Lee et al., 2022) or retrain the whole model
(Park et al., 2020; Pan and Kang, 2021), which is time-consuming and complex
in practical applications. To address this issue, the proposed method only
uses the original features of the unseen nodes (i.e., $\textbf{X}_{un}$) to
infer their representations, which is efficient and simple. Specifically,
after the whole network (i.e., $\\{\mathcal{F}_{\Theta^{v}}^{v}\\}_{v=1}^{V}$)
is trained, we directly apply each MLP encoder to obtain the representations
for unseen nodes as follows:

(3) $\mathbf{Z}^{v}_{un}=\mathcal{F}_{\Theta^{v}}^{v}(\mathbf{X}_{un}),$

where the representations will be fused by the average operation to obtain the
final representations for unseen nodes. In this way, we only use the original
feature information in the inference process, and our method achieves
efficient and simple inference for unseen nodes without constructing new
matrices or retraining. Despite its efficiency, our method is also able to
obtain effective representations for unseen nodes because it can implicitly
learn the mapping from features to structures by Eq. (2), which can preserve
the local structure among the training nodes and unseen nodes.

The advantages of our proposed complementary information extraction are listed
as follows. First, we employ the MLP as encoders instead of GCN, which makes
representation learning more efficient. Second, the representations
characterize the graph structure by satisfying the aforementioned three
conditions, making them effective. Third, the MLP encoders implicitly learn
the mappings from the features to graph structure, enabling them to solve the
out-of-sample issue. Consequently, CoCoMG is effective and efficient in
exploring complementary information while also addressing the out-of-sample
issue.

###  2.3. Consistent information extraction

Although node representations on multiple graphs provide complementary
information, they can also contain noisy information that decreases the
model’s robustness. To reduce the impact of noise, recent studies apply
contrastive learning to extract the consistent information across multiple
graphs, but they easily result in inefficiency and ineffectiveness.

In this paper, we propose to effectively and efficiently exploit the
consistent information by directly maximizing the correlation among the
representations of each graph based on the following conditions: (i)
Correlation. Enforcing consistent node representations across different
graphs, as canonical correlation analysis (i.e., CCA) does (Hardoon et al.,
2004); (ii) Decorrelation. Enforcing the disagreement of node representation
in each graph to maintain discriminative characteristics (Jing et al., 2022b).

To do this, given the representation matrices $\mathbf{Z}^{v}$ ($v=1,...,V$),
we maximizes the correlation coefficient $\rho$ between two representation
matrices $\mathbf{Z}^{a}$ and $\mathbf{Z}^{b}$, i.e.,

(4)
$\rho\left(\mathbf{Z}^{a},\mathbf{Z}^{b}\right)=\frac{(\mathbf{Z}^{a})^{\top}\mathbf{Z}^{b}}{\sqrt{\left((\mathbf{Z}^{a})^{\top}\mathbf{Z}^{a}\right)\left((\mathbf{Z}^{b})^{\top}\mathbf{Z}^{b}\right)}}.$

After replacing the variance item $(\mathbf{Z}^{a})^{\top}\mathbf{Z}^{a}$ with
the constrain
$\Sigma_{\mathbf{Z}^{a},\mathbf{Z}^{a}}=\Sigma_{\mathbf{Z}^{b},\mathbf{Z}^{b}}=\mathbf{I}$,
we define the CCA regularization loss for UMGL as:

(5)
$\mathcal{L}_{\text{CCA}}=-\sum_{a=1}^{V}\sum_{b>a}^{V}\mathbf{Z}^{a}(\mathbf{Z}^{b})^{\top}+\gamma\sum_{v=1}^{V}\left\|\left(\mathbf{Z}^{v}\right)^{\top}\mathbf{Z}^{v}-\mathbf{I}\right\|_{F}^{2},$

where $\gamma$ is the trade-off coefficient. In Eq. (5), the first term
encourages the correlation among node representations across different graphs.
The second term maintains the decorrelation among node representations in each
graph, which ensures that individual dimensions of the learned representation
are uncorrelated to avoid trivial solutions.

Tackling the noise issue. One of the challenges for UMGL is to deal with noisy
structures in multiplex graph, where the qualities of different graphs
captured by various sensors cannot be guaranteed, e.g., the graph structures
from some sensors may be unreliable and bring noisy information. Previous UMGL
methods (Park et al., 2020; Jing et al., 2021; Wang et al., 2021; Pan and
Kang, 2021; Lee et al., 2022) first use GCN on each graph to obtain multiple
representations and then apply contrastive learning to maintain the
consistency among multiple representations for noise resistance, which is both
ineffective and inefficient. Differently, the proposed method first obtains
multiple representations by the MLP encoders, which means the noisy
information will not be directly introduced into multiple representations.
Note that, previous UMGL methods need to aggregate representation from
neighborhoods, which will directly incorporate the noisy information into
multiple representations. We then apply the CCA loss Eq. (5) to directly
maintain the consistency among multiple representations without negative
pairs, which can resist the noisy influence from noisy graphs by balancing the
local preserve loss in Eq. (2). Specifically, if the view-specific
representation preserves the local structure of the noisy graph, i.e.,
minimizing Eq. (2), the obtained representations are inconsistent across
multiple graphs. As a result, the learnable model will be heavily penalized by
Eq. (5).

The advantages of our proposed consistent information extraction are listed as
follows. First, the proposed method can effectively deal with the noise issue,
as the noisy information is not directly incorporated into the
representations. Moreover, the noisy information in Eq. (2) can be corrected
and reduced by Eq. (5). Second, the proposed method is more efficient than
previous methods, as it does not need to construct positive and negative pairs
(verified in Section 3.4).

###  2.4. Loss function

Integrating Eq. (2) with Eq. (5), the final objective function of CoCoMG is:

(6) $\mathcal{L}=\mathcal{L}_{\text{LP}}+\beta\mathcal{L}_{\text{CCA}},$

where $\beta$ achieves the trade-off between $\mathcal{L}_{\text{LP}}$ and
$\mathcal{L}_{\text{CCA}}$. In the optimization of CoCoMG,
$\mathcal{L}_{\text{LP}}$ explores the complementary information for every
graph, and $\mathcal{L}_{\text{CCA}}$ explores the consistent information
across multiple graphs. Subsequently, we fuse the representation for multiple
graphs by the average operation to obtain the final representation (i.e.,
$\mathbf{Z}=\frac{1}{V}\sum_{v=1}^{V}\mathbf{Z}^{v}$), which is further used
for downstream tasks.

Table 1. Classification performance (i.e., Macro-F1 and Micro-F1) of all methods on all multiplex graph datasets. Method | ACM | IMDB | DBLP | Freebase
---|---|---|---|---
Macro-F1 | Micro-F1 | Macro-F1 | Micro-F1 | Macro-F1 | Micro-F1 | Macro-F1 | Micro-F1
Deep Walk | 73.9 $\pm$ 0.2 | 74.8 $\pm$ 0.2 | 42.5 $\pm$ 0.2 | 43.3 $\pm$ 0.4 | 88.1 $\pm$ 0.2 | 89.5 $\pm$ 0.3 | 49.3 $\pm$ 0.3 | 52.1 $\pm$ 0.2
GCN | 86.9 $\pm$ 0.3 | 87.0 $\pm$ 0.2 | 45.7 $\pm$ 0.4 | 49.8 $\pm$ 0.2 | 90.2 $\pm$ 0.2 | 90.9 $\pm$ 0.2 | 50.5 $\pm$ 0.2 | 53.3 $\pm$ 0.2
GAT | 85.0 $\pm$ 0.2 | 84.9 $\pm$ 0.1 | 49.4 $\pm$ 0.2 | 53.6 $\pm$ 0.4 | 91.0 $\pm$ 0.4 | 92.1 $\pm$ 0.2 | 55.1 $\pm$ 0.3 | 59.7 $\pm$ 0.4
DGI | 88.1 $\pm$ 0.5 | 88.1 $\pm$ 0.4 | 45.1 $\pm$ 0.2 | 46.7 $\pm$ 0.2 | 90.3 $\pm$ 0.1 | 91.1 $\pm$ 0.4 | 54.9 $\pm$ 0.1 | 58.2 $\pm$ 0.2
MNE | 79.2 $\pm$ 0.4 | 79.7 $\pm$ 0.2 | 44.7 $\pm$ 0.5 | 45.6 $\pm$ 0.3 | 89.3 $\pm$ 0.2 | 90.6 $\pm$ 0.4 | 52.1 $\pm$ 0.3 | 54.3 $\pm$ 0.2
DMGI | 89.8 $\pm$ 0.2 | 89.8 $\pm$ 0.2 | 52.2 $\pm$ 0.2 | 53.7 $\pm$ 0.3 | 92.1 $\pm$ 0.2 | 92.9 $\pm$ 0.3 | 54.9 $\pm$ 0.1 | 57.6 $\pm$ 0.2
DMGIattn | 88.7 $\pm$ 0.2 | 88.7 $\pm$ 0.2 | 52.6 $\pm$ 0.2 | 53.6 $\pm$ 0.4 | 90.9 $\pm$ 0.2 | 91.8 $\pm$ 0.3 | 55.8 $\pm$ 0.4 | 58.3 $\pm$ 0.1
HDMI | 90.1 $\pm$ 0.3 | 90.1 $\pm$ 0.1 | 55.6 $\pm$ 0.3 | 57.3 $\pm$ 0.3 | 91.3 $\pm$ 0.2 | 92.2 $\pm$ 0.5 | 56.1 $\pm$ 0.2 | 59.2 $\pm$ 0.2
HeCo | 88.2 $\pm$ 0.2 | 88.3 $\pm$ 0.3 | 50.8 $\pm$ 0.3 | 51.7 $\pm$ 0.3 | 91.0 $\pm$ 0.3 | 91.6 $\pm$ 0.3 | 59.2 $\pm$ 0.3 | 61.7 $\pm$ 0.2
MCGC | 90.2 $\pm$ 0.8 | 90.0 $\pm$ 0.1 | 56.3 $\pm$ 0.5 | 57.5 $\pm$ 0.6 | 91.9 $\pm$ 0.3 | 92.1 $\pm$ 0.4 | 56.6 $\pm$ 0.1 | 59.4 $\pm$ 0.1
CKD | 90.4 $\pm$ 0.3 | 90.5 $\pm$ 0.2 | 54.8 $\pm$ 0.2 | 57.7 $\pm$ 0.3 | 92.0 $\pm$ 0.2 | 92.3 $\pm$ 0.5 | 60.4 $\pm$ 0.4 | 62.9 $\pm$ 0.4
RGRL | 90.3 $\pm$ 0.4 | 90.2 $\pm$ 0.4 | 52.1 $\pm$ 0.2 | 55.5 $\pm$ 0.5 | 91.7 $\pm$ 0.2 | 92.0 $\pm$ 0.3 | 59.4 $\pm$ 0.4 | 62.1 $\pm$ 0.3
CoCoMG | 92.8 $\pm$ 0.1 | 92.7 $\pm$ 0.1 | 58.3 $\pm$ 0.4 | 59.6 $\pm$ 0.4 | 92.1 $\pm$ 0.1 | 92.9 $\pm$ 0.1 | 60.8 $\pm$ 0.1 | 63.8 $\pm$ 0.9
Table 2. Clustering performance (i.e., Accuracy and NMI) of unsupervised methods on all multiplex graph datasets. Method | ACM | IMDB | DBLP | Freebase
---|---|---|---|---
Accuracy | NMI | Accuracy | NMI | Accuracy | NMI | Accuracy | NMI
Deep Walk | 64.5 $\pm$ 0.7 | 41.6 $\pm$ 0.5 | 42.1 $\pm$ 0.4 | 1.5 $\pm$ 0.1 | 89.5 $\pm$ 0.4 | 69.0 $\pm$ 0.2 | 44.5 $\pm$ 0.6 | 12.8 $\pm$ 0.4
DGI | 81.1 $\pm$ 0.6 | 64.0 $\pm$ 0.4 | 48.9 $\pm$ 0.2 | 8.3 $\pm$ 0.3 | 85.4 $\pm$ 0.3 | 65.6 $\pm$ 0.4 | 52.9 $\pm$ 0.2 | 17.8 $\pm$ 0.2
MNE | 69.1 $\pm$ 0.2 | 54.5 $\pm$ 0.3 | 46.5 $\pm$ 0.3 | 4.6 $\pm$ 0.2 | 86.3 $\pm$ 0.3 | 68.4 $\pm$ 0.2 | 45.1 $\pm$ 0.5 | 13.3 $\pm$ 0.7
DMGI | 88.4 $\pm$ 0.3 | 68.7 $\pm$ 0.5 | 52.5 $\pm$ 0.7 | 13.1 $\pm$ 0.3 | 91.8 $\pm$ 0.5 | 76.4 $\pm$ 0.6 | 53.1 $\pm$ 0.4 | 17.3 $\pm$ 0.4
DMGIattn | 90.9 $\pm$ 0.4 | 70.2 $\pm$ 0.3 | 52.6 $\pm$ 0.3 | 9.2 $\pm$ 0.2 | 91.3 $\pm$ 0.4 | 75.2 $\pm$ 0.4 | 52.3 $\pm$ 0.5 | 17.1 $\pm$ 0.3
HDMI | 90.8 $\pm$ 0.4 | 69.5 $\pm$ 0.5 | 57.6 $\pm$ 0.4 | 14.5 $\pm$ 0.4 | 90.1 $\pm$ 0.4 | 73.1 $\pm$ 0.3 | 58.3 $\pm$ 0.3 | 20.3 $\pm$ 0.4
HeCo | 88.4 $\pm$ 0.6 | 67.8 $\pm$ 0.8 | 50.9 $\pm$ 0.5 | 10.1 $\pm$ 0.6 | 89.2 $\pm$ 0.3 | 71.0 $\pm$ 0.7 | 58.4 $\pm$ 0.6 | 20.4 $\pm$ 0.5
MCGC | 90.4 $\pm$ 0.5 | 69.0 $\pm$ 0.5 | 56.5 $\pm$ 0.3 | 14.9 $\pm$ 0.4 | 91.9 $\pm$ 0.2 | 76.5 $\pm$ 0.4 | 58.1 $\pm$ 0.4 | 20.2 $\pm$ 0.3
CKD | 90.6 $\pm$ 0.4 | 69.3 $\pm$ 0.3 | 53.9 $\pm$ 0.3 | 13.8 $\pm$ 0.4 | 91.4 $\pm$ 0.4 | 75.9 $\pm$ 0.4 | 58.5 $\pm$ 0.6 | 20.6 $\pm$ 0.4
RGRL | 90.7 $\pm$ 0.5 | 69.4 $\pm$ 0.2 | 52.5 $\pm$ 0.5 | 13.3 $\pm$ 0.2 | 91.0 $\pm$ 0.3 | 74.5 $\pm$ 0.5 | 58.9 $\pm$ 0.3 | 20.8 $\pm$ 0.4
CoCoMG | 92.3 $\pm$ 0.1 | 73.6 $\pm$ 0.1 | 59.0 $\pm$ 0.4 | 17.4 $\pm$ 0.2 | 91.3 $\pm$ 0.2 | 75.6 $\pm$ 0.3 | 64.0 $\pm$ 0.1 | 24.1 $\pm$ 0.1

Figure 2. Classification performance of our method and four UMGL methods for
the out-of-sample extension on all multiplex graph datasets. The “ratio”
indicates the percentage of unseen nodes in each dataset.

Figure 3. Inference time (seconds) of CoCoMG and four UMGL methods for the
out-of-sample extension on all datasets.

Figure 4. Classification performance of our method and all self-supervised
multiplex graph representation methods under different noisy edges ratios
$\eta$ on all multiplex graph datasets.

##  3\. Experiments

###  3.1. Experimental Setup

####  3.1.1. Datasets

We use four public benchmark datasets to evaluate the performance of our
proposed method, including two citation multiplex graph networks (i.e., ACM
(Wang et al., 2019) and DBLP (Park et al., 2020)), and two movie multiplex
graph networks (i.e., IMDB (Park et al., 2020) and Freebase (Jing et al.,
2021)). We follow the public splitting settings (Jing et al., 2021; Wang et
al., 2021) in our experiments.

####  3.1.2. Comparison methods

The comparison methods include four single-view graph learning methods and
eight multiplex graph learning methods. The single-view graph learning methods
consist of two semi-supervised methods, i.e., GCN (Kipf and Welling, 2017),
and GAT (Velickovic et al., 2018), one traditional unsupervised learning
method, i.e., DeepWalk (Perozzi et al., 2014), and one self-supervised
learning method, i.e., DGI (Velickovic et al., 2019). The multiplex graph
learning methods include one traditional unsupervised learning method,i.e.,
MNE (Zhang et al., 2018), and seven self-supervised learning methods, i.e.,
DMGI (Park et al., 2020), DMGIattn (Park et al., 2020), HDMI (Jing et al.,
2021), HeCo (Wang et al., 2021), MCGC (Pan and Kang, 2021), CKD (Wang et al.,
2022c), and RGRL (Lee et al., 2022). To ensure a fair comparison, we evaluate
single-view graph methods on multiplex graph datasets by training them
independently on each graph and then concatenating their learned
representations for downstream tasks. The average values of 10 runs are
reported.

####  3.1.3. Evaluation protocol

We follow the evaluation in previous works (Jing et al., 2021; Wang et al.,
2022c) to conduct node classification and node clustering as semi-supervised
and unsupervised downstream tasks, respectively. Specifically, we employ
Macro-F1 and Micro-F1 to evaluate the performance of node classification.
Moreover, we employ Accuracy and Normalized Mutual Information (NMI) to
evaluate the performance of node clustering. Additionally, to evaluate the
robustness of our method and comparison methods, we randomly replace a certain
ratio of edges in each graph with noisy edges (i.e., random edges).

###  3.2. Effectiveness Analysis

To evaluate the effectiveness of the proposed method, we report the
performance of node classification (i.e., Macro-F1 and Micro-F1) and node
clustering (i.e., Accuracy and NMI) of all methods in Table 1 and 2,
respectively. Obviously, the proposed method obtains the highest effectiveness
on both node classification and node clustering tasks.

First, compared to the single graph learning methods (i.e., DeepWalk, GCN,
GAT, and DGI), the proposed CoCoMG outperforms them by significant margins.
For example, CoCoMG achieves an average improvement of 11.4% and 29.7% in
terms of classification and clustering tasks, respectively, compared to the
best-performing unsupervised single graph method DGI. Moreover, most multiplex
graph learning methods have achieved better performance than single graph
methods on both node classification and clustering tasks This suggests that
multiplex graph learning methods are more effective than single-view graph
methods, as they can explore the complementary and consistent information
among different graphs to learn more discriminative node representations.

Second, compared to the multiplex graph learning methods, the proposed CoCoMG
achieved the best performance, followed by CKD, MCGC, RGRL, HDMI, HeCo, DMGI,
DMGIattn, and MNE. The reason can be attributed to that the proposed method
can effectively extract complementary and consistent information while
reducing the impact of noise.

###  3.3. Out-of-sample Analysis

In this section, we compare the effectiveness and efficiency of the proposed
method with previous UMGL methods when tackling the out-of-sample issue. Note
that, the comparison method, such as DMGI (Park et al., 2020), DMGIattn (Park
et al., 2020), and MCGC (Pan and Kang, 2021), cannot handle the out-of-sample
issue as they need to retrain the whole model to infer the representation of
the unseen nodes. Thus, we only compare our method with HDMI (Jing et al.,
2021), HeCo (Wang et al., 2021), CKD (Wang et al., 2022c), and RGRL (Lee et
al., 2022) on all multiplex graph datasets. To do this, we first randomly
split (repeated 5 times with random seeds) the unseen nodes from all nodes in
the dataset with different ratios (i.e., 0 and 0.4) for each dataset. We then
train the model for each method on the in-sample nodes with their local graph
structures. After the training is completed, we use the optimized model to
infer the representation of unseen nodes and evaluate the inference time and
the performance on the node classification task.

The performance results are shown in Figure 2. Firstly, in terms of horizontal
comparison (i.e., blue bars of all methods), the proposed method achieves the
best performance on all datasets compared to the comparison methods, followed
by HDMI, RGRL, CKD, and HeCo. Secondly, in terms of vertical comparison (i.e.,
the performance gap between red bars and blue bars), we observe that the
performance drop of our method compared to the comparison methods is the least
significant. The reason can be that previous methods only preserve the graph
structure without learning an inference model that can map the input to the
graph structure. As a result, these methods lack the inference ability when
facing unseen nodes and their local graph structures. In contrast, our method
learns an inference model that can implicitly map features to the graph
structure, allowing it to effectively infer the representations of unseen
nodes by implicitly mapping them to the graph structures among the seen nodes.

The inference times of our proposed method and comparison methods are
presented in Figure 3, demonstrating the superior efficiency of our approach.
Across all multiplex graph datasets, our method is on average 33.1× and 8.3×
faster than the slowest method RGRL and the fastest method HDMI, respectively.
This can be attributed to the fact that our method only utilizes MLP encoders
for representation inference. These results are promising, as they indicate
that our proposed method is capable of effectively tackling the out-of-sample
issue while achieving superior efficiency.

Table 3. Classification performance (i.e., Macro-F1 and Micro-F1) of each constraint in CoCoMG on all mutiplex graph datasets. |  | ACM | IMDB | DBLP | Freebase
---|---|---|---|---|---
$\mathcal{L}_{LP}$ | $\mathcal{L}_{CCA}$ | Macro-F1 | Micro-F1 | Macro-F1 | Micro-F1 | Macro-F1 | Micro-F1 | Macro-F1 | Micro-F1
$-$ | $\surd$ | 86.8 $\pm$ 0.8 | 86.9 $\pm$ 0.7 | 51.0 $\pm$ 0.4 | 51.6 $\pm$ 0.8 | 53.2 $\pm$ 0.2 | 77.9 $\pm$ 0.2 | 35.3 $\pm$ 0.3 | 42.7 $\pm$ 0.8
$\surd$ | $-$ | 92.1 $\pm$ 0.2 | 92.1 $\pm$ 0.1 | 51.5 $\pm$ 0.6 | 52.1 $\pm$ 0.8 | 90.5 $\pm$ 0.4 | 91.5 $\pm$ 0.4 | 59.0 $\pm$ 0.5 | 62.4 $\pm$ 0.1
$\surd$ | $\surd$ | 92.8 $\pm$ 0.1 | 92.7 $\pm$ 0.1 | 58.3 $\pm$ 0.4 | 59.6 $\pm$ 0.4 | 92.1 $\pm$ 0.1 | 92.9 $\pm$ 0.1 | 60.8 $\pm$ 0.1 | 63.8 $\pm$ 0.9

(a) w/o $\mathcal{L}_{LP}$(SIL=0.42)

(b) w/o $\mathcal{L}_{CCA}$(SIL=0.53)

(c) with $\mathcal{J}$(SIL=0.74)

Figure 5. Visualization plotted by $t$-SNE and the corresponding silhouette
scores (SIL) of node representations for our method on Dataset ACM, where
different colors represent different classes of the nodes. (a) CoCoMG without
$\mathcal{L}_{LP}$; (b) CoCoMG without $\mathcal{L}_{CCA}$; (c) CoCoMG with
two constraints.

###  3.4. Robustness Analysis

To verify the robustness of the proposed method, we report the performance of
all self-supervised multiplex graph learning methods on the node
classification task under varying ratios of noisy edges (random edges) ranging
from 0.1 to 0.9. These results are presented in Figure 4.

From Figure 4, we have the observations as follows. Firstly, the proposed
method consistently achieves the best performance on all datasets under
different noise ratios, confirming our claim that CoCoMG obtains satisfying
prediction performance under noise. Secondly, as the noise ratio increases,
the performance of all methods decreases, while the proposed method exhibits
the smallest decrease. For instance, on Dataset ACM, the proposed method is
minimally affected in terms of performance when the noise ratio is below 0.7.
Even when the noise ratio reaches 0.9, the performance only decreases by
approximately 6%, while other methods show a decrease of over 40%. The reasons
behind these results are twofold. Firstly, we only use the MLP encoders to
obtain node representations, which means noise cannot be directly aggregated
into the node representations through incorrect edges. Secondly, we use Eq.
(5) to ensure the consistency of the representations between different views,
which reduces the noise caused by Eq. (2).

###  3.5. Ablation Study

The proposed method employs two constraints (i.e., $\mathcal{L}_{LP}$ in Eq.
(2) and $\mathcal{L}_{CCA}$ in Eq. (5)) to extract complementary and
consistent information, respectively. To verify the effectiveness of each
constraint of the proposed method, we investigate the performance(i.e.,
Macro-F1 and Micro-F1) of each constraint on the node classification task, and
report the results in Table 3. Generally, we observe that the proposed method
with two constraints achieves the best performance. Conversely, when we only
implement an individual constraint (i.e., either $\mathcal{L}_{LP}$ or
$\mathcal{L}_{CCA}$), the performance is inferior. Moreover, as shown in
Figure 5, we utilize $t$-SNE (Van der Maaten and Hinton, 2008) to visualize
the node representations learned with different constraints on the ACM
dataset. We observed that the representations of nodes belonging to the same
class, learned with both constraints, are closer to each other than the
representations learned with only one constraint. Consequently, the
experiments confirm that both constraints play a crucial role in the
effectiveness of CoCoMG.

###  3.6. Over-smooth Analysis

Previous UMGL methods have relied heavily on GCN to obtain node embeddings.
However, as the number of layers in a GCN increases, the performance of the
model tends to decrease, which is known as the over-smooth problem. In fact,
modern deep neural networks typically require multiple layers to effectively
extract features and learn complex patterns, but in graph neural networks,
using too many graph convolutional layers can lead to the over-smooth
phenomenon, which limits the number of layers in a GCN. To some extent, this
presents a paradoxical situation in which the depth of the GCN is both
necessary and limited.

In this section, we analyze the effectiveness of the proposed method when
dealing with the over-smooth problem. For this purpose, we evaluate the
effectiveness of our method and vanilla GCN (i.e., using GCN as encoder) with
varying numbers of layers ranging from $1$ to $19$ on Dataset ACM, and the
results are presented in Figure 6. Overall, the performance of our method
shows a slight increase and remains stable with the number of layers
increasing, while vanilla GCN’s performance drops sharply. Specifically, our
model’s performance improves as the number of layers increases from $1$ to
$11$, and the performance of our model remains stable with the increase in the
number of layers from $11$ to $19$. In contrast, for vanilla GCN, the
performance drops significantly with the number of layers from $1$ to $19$,
with a drop of over 60%. The reason might be as follows. For our model, as the
number of layers increases, the model can learn more abstract node embeddings,
and thus the performance of the model increases gradually. For vanilla GCN,
however, the message passing mechanism causes the node embeddings to become
more and more similar as the layers increase, significantly degrading
performance. Based on this observation, we can conclude that our method can
effectively address the over-smoothing problem.

Figure 6. Classification results of our method and vanilla GCN under different
number of layers on Dataset ACM

###  3.7. Parameter Sensitivity

The hyper-parameter of CoCoMG includes the trade-off $\gamma$ in Eq. (5) and
$\beta$ in Eq. (6), and Figure 7 shows the effectiveness by traversing
$\gamma$ and $\beta$ on Dataset ACM. The results indicate that $\beta$ is
relatively insensitive in the range of 0.01 to 10, while $\gamma$ exhibits
little sensitivity to performance. These results suggest that CoCoMG can
achieve near-perfect performance when the value of $\gamma$ is in a reasonable
range (e.g., $\gamma>0.1$).

Figure 7. Classification results of our method at different parameter settings
(i.e., $\gamma$ and $\beta$) on Dataset ACM.

##  4\. Conclusion

In this paper, we investigated two pervasive but challenging issues of
existing UMGL methods, i.e., the out-of-sample issue and the noise issue. To
tackle these issues, we proposed CoCoMG, that only employs multiple MLP
encoders with two constraints to effectively and efficiently extract
complementary and consistent information for conducting unsupervised multiple
graph learning. Comprehensive experiments demonstrate that CoCoMG achieves
state-of-the-art performance in terms of both effectiveness and efficiency, as
well as confirm our claim that CoCoMG can tackle those two issues.

## References

  * (1)
  * Battiston et al. (2017) Federico Battiston, Vincenzo Nicosia, and Vito Latora. 2017\.  The new challenges of multiplex networks: Measures and models.  _The European Physical Journal Special Topics_ 226 (2017), 401–416.
  * Cen et al. (2019) Yukuo Cen, Xu Zou, Jianwei Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. 2019\.  Representation learning for attributed multiplex heterogeneous network. In _KDD_. 1358–1368.
  * Chu et al. (2019) Xiaokai Chu, Xinxin Fan, Di Yao, Zhihua Zhu, Jianhui Huang, and Jingping Bi. 2019\.  Cross-Network Embedding for Multi-Network Alignment. In _WWW_. 273–284.
  * Dong et al. (2016) Xiaowen Dong, Dorina Thanou, Pascal Frossard, and Pierre Vandergheynst. 2016.  Learning Laplacian Matrix in Smooth Graph Signal Representations.  _IEEE Transactions on Signal Process._ 64, 23 (2016), 6160–6173.
  * Gan et al. (2022) Jiangzhang Gan, Rongyao Hu, Yujie Mo, Zhao Kang, Liang Peng, Yonghua Zhu, and Xiaofeng Zhu. 2022.  Multi-graph Fusion for Dynamic Graph Convolutional Network.  _IEEE Transactions on Neural Networks and Learning Systems_ (2022), 10.1109/TNNLS.2022.3172588.
  * Hamilton et al. (2017) William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017\.  Inductive Representation Learning on Large Graphs. In _NeurIPS_. 1024–1034.
  * Han et al. (2023) Beibei Han, Yingmei Wei, Qingyong Wang, and Shanshan Wan. 2023\.  CoLM2S: Contrastive self-supervised learning on attributed multiplex graph network with multi-scale information.  _CAAI Transactions on Intelligence Technology_ (2023).
  * Hardoon et al. (2004) David R. Hardoon, Sándor Szedmák, and John Shawe-Taylor. 2004.  Canonical correlation analysis: An overview with application to learning methods.  _Neural Computing._ 16, 12 (2004), 2639–2664.
  * Jing et al. (2022a) Baoyu Jing, Shengyu Feng, Yuejia Xiang, Xi Chen, Yu Chen, and Hanghang Tong. 2022a.  X-GOAL: Multiplex Heterogeneous Graph Prototypical Contrastive Learning. In _CIKM_. 894–904.
  * Jing et al. (2021) Baoyu Jing, Chanyoung Park, and Hanghang Tong. 2021\.  HDMI: High-order Deep Multiplex Infomax. In _WWW_. 2414–2424.
  * Jing et al. (2022b) Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. 2022b.  Understanding Dimensional Collapse in Contrastive Self-supervised Learning. In _ICLR_.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Variational graph auto-encoders.  _arXiv preprint arXiv:1611.07308_ (2016).
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _ICLR_. 1–14.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015\.  Deep learning.  _Nature_ 521, 7553 (2015), 436–444.
  * Lee et al. (2022) Namkyeong Lee, Dongmin Hyun, Junseok Lee, and Chanyoung Park. 2022.  Relational Self-Supervised Learning on Graphs. In _CIKM_. 1054–1063.
  * Li et al. (2021) Zhiqiang Li, Hao Wang, and Jian Zhang. 2021.  A multiplex graph learning framework for predicting drug-disease associations.  _IEEE/ACM Transactions on Computational Biology and Bioinformatics_ 18, 3 (2021), 807–818.
  * Liu et al. (2022) Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S Yu. 2022\.  Graph self-supervised learning: A survey.  _IEEE Transactions on Knowledge and Data Engineering_ (2022), 1–1.
  * Ma et al. (2018) Jianxin Ma, Peng Cui, and Wenwu Zhu. 2018.  DepthLGP: Learning Embeddings of Out-of-Sample Nodes in Dynamic Networks. In _AAAI_. 370–377.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Pan and Kang (2021) Erlin Pan and Zhao Kang. 2021.  Multi-view Contrastive Graph Clustering. In _NeurIPS_. 2148–2159.
  * Park et al. (2020) Chanyoung Park, Donghyun Kim, Jiawei Han, and Hwanjo Yu. 2020\.  Unsupervised Attributed Multiplex Network Embedding. In _AAAI_. 5371–5378.
  * Peng et al. (2022) Liang Peng, Rongyao Hu, Fei Kong, Jiangzhang Gan, Yujie Mo, Xiaoshuang Shi, and Xiaofeng Zhu. 2022.  Reverse Graph Learning for Graph Neural Network.  _IEEE Transactions on Neural Networks and Learning Systems_ (2022), 1–12.  https://doi.org/10.1109/TNNLS.2022.3161030
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014\.  Deepwalk: Online learning of social representations. In _KDD_. 701–710.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ 9, 11 (2008).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _NeurIPS_ , Vol. 30.
  * Veličković (2022) Petar Veličković. 2022\.  Message passing all the way up.  _arXiv preprint arXiv:2202.11097_ (2022).
  * Velickovic et al. (2018) Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks. In _ICLR_. 1–12.
  * Velickovic et al. (2019) Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. 2019.  Deep Graph Infomax. In _ICLR_. 1–17.
  * Wang et al. (2022c) Can Wang, Sheng Zhou, Kang Yu, Defang Chen, Bolang Li, Yan Feng, and Chun Chen. 2022c.  Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding. In _WWW_. 1631–1639.
  * Wang et al. (2022a) Haoqing Wang, Xun Guo, Zhi-Hong Deng, and Yan Lu. 2022a.  Rethinking Minimal Sufficient Representation in Contrastive Learning. In _CVPR_. IEEE, 16020–16029.
  * Wang et al. (2022b) Qiang Wang, Hao Jiang, Ying Jiang, Shuwen Yi, Qi Nie, and Geng Zhang. 2022b.  Multiplex network infomax: Multiplex network embedding via information fusion.  _Digital Communications and Networks_ (2022).
  * Wang et al. (2019) Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S. Yu. 2019.  Heterogeneous Graph Attention Network. In _WWW_. 2022–2032.
  * Wang et al. (2021) Xiao Wang, Nian Liu, Hui Han, and Chuan Shi. 2021\.  Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning. In _KDD_. 1726–1736.
  * Xie et al. (2023) Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. 2023.  Self-Supervised Learning of Graph Neural Networks: A Unified Review.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 45, 2 (2023), 2412–2429.
  * Yuan et al. (2021) Changan Yuan, Zhi Zhong, Cong Lei, Xiaofeng Zhu, and Rongyao Hu. 2021.  Adaptive reverse graph learning for robust subspace learning.  _Information Processing & Management_ 58, 6 (2021), 102733\.
  * Zhang et al. (2018) Hongming Zhang, Liwei Qiu, Lingling Yi, and Yangqiu Song. 2018\.  Scalable Multiplex Network Embedding. In _IJCAI_. 3082–3088.
  * Zhang et al. (2019) Jun Zhang, Peng Cui, Xiao Wang, and Wenwu Zhu. 2019\.  Multiplex graph learning: Algorithms and applications.  _IEEE Transactions on Neural Networks and Learning Systems_ 30, 1 (2019), 14–28.
  * Zhang et al. (2023) Qiqi Zhang, Zhongying Zhao, Hui Zhou, Xiangju Li, and Chao Li. 2023.  Self-supervised Contrastive Learning on Heterogeneous Graphs with Mutual Constraints of Structure and Feature.  _Information Sciences_ (2023), 119026.
  * Zhang et al. (2022) Rui Zhang, Arthur Zimek, and Peter Schneider-Kamp. 2022\.  Unsupervised Representation Learning on Attributed Multiplex Network. In _CIKM_. 2610–2619.
  * Zhao et al. (2017) Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. 2017\.  Multi-view learning overview: Recent progress and new challenges.  _Information Fusion_ 38 (2017), 43–54.
  * Zhou et al. (2020) Daokun Zhou, Wei Bao, and Wei Liu. 2020.  Multiple graph learning: An unsupervised feature learning approach for graph-structured data.  _IEEE Transactions on Neural Networks and Learning Systems_ 31, 7 (2020), 2632–2642.
  * Zhu et al. (2018) Dingyuan Zhu, Peng Cui, Ziwei Zhang, Jian Pei, and Wenwu Zhu. 2018.  High-order proximity preserved embedding for dynamic networks.  _IEEE Transactions on Knowledge and Data Engineering_ 30, 11 (2018), 2134–2144.
  * Zhu et al. (2022a) Yonghua Zhu, Junbo Ma, Changan Yuan, and Xiaofeng Zhu. 2022a.  Interpretable learning based Dynamic Graph Convolutional Networks for Alzheimer’s Disease analysis.  _Information Fusion_ 77 (2022), 53–61.
  * Zhu et al. (2022b) Yanqiao Zhu, Yichen Xu, Hejie Cui, Carl Yang, Qiang Liu, and Shu Wu. 2022b.  Structure-Enhanced Heterogeneous Graph Contrastive Learning. In _ICDM_. 82–90.
  * Zhu et al. (2021) Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2021.  Graph contrastive learning with adaptive augmentation. In _WWW_. 2069–2080.