# Few Shot Generative Model Adaption via Relaxed Spatial Structural Alignment

Jiayu Xiao1,2, Liang Li1, Chaofei Wang3, Zheng-Jun Zha4, Qingming Huang1,2,5
1Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS, Beijing,
China
2University of Chinese Academy of Sciences, Beijing, China, 3Department of
Automation, Tsinghua University
4University of Science and Technology of China, China, 5Peng Cheng Laboratory,
Shenzhen, China
jiayu.xiao<EMAIL_ADDRESS>
wangcf18@mails.tsinghua.edu.cn,zhazj<EMAIL_ADDRESS>Corresponding author.Equal contribution.

###### Abstract

Training a generative adversarial network (GAN) with limited data has been a
challenging task. A feasible solution is to start with a GAN well-trained on a
large scale source domain and adapt it to the target domain with a few
samples, termed as few shot generative model adaption. However, existing
methods are prone to model overfitting and collapse in extremely few shot
setting (less than 10). To solve this problem, we propose a relaxed spatial
structural alignment (RSSA) method to calibrate the target generative models
during the adaption. We design a cross-domain spatial structural consistency
loss comprising the self-correlation and disturbance correlation consistency
loss. It helps align the spatial structural information between the synthesis
image pairs of the source and target domains. To relax the cross-domain
alignment, we compress the original latent space of generative models to a
subspace. Image pairs generated from the subspace are pulled closer.
Qualitative and quantitative experiments show that our method consistently
surpasses the state-of-the-art methods in few shot setting. Our source code:
https://github.com/StevenShaw1999/RSSA.

##  1 Introduction

Figure 1: Few shot generative model adaption and our motivation. Start with a
pre-trained model of the source domain $G_{s}$ and adapt to the target domain
to get $G_{t}$ by using extremely few (such as 5) training images of the
target domain. We compress the latent space to a subspace close to target
domain, and align the spatial structural information of synthesis pairs
generated by $G_{s}$ and $G_{t}$.

Generative adversarial networks (GANs) have achieved promising results in
various computer vision scenarios such as natural image synthesis [3, 10],
image to image translation[42] and image inpainting[35, 38]. Meanwhile, GANs
are notoriously hard to train, and training an image generative model
generally requires thousands of images and tens of hours of training time.
Actually, for many real-world applications, data acquisition is difficult or
expensive. For example, in the artistic domain, it is impossible to hire
artists to make thousands of creations. Without enough training data, GANs are
prone to overfit and collapse.

To address this issue, researchers begin to focus on effective GAN training
with limited samples. Most of them follow the route of few shot generative
model adaption that starting with a model pre-trained on a large dataset of a
source domain, and adapting to the target domain with limited data, as shown
in Fig. 1. Wang _et al_. [32] leverage the fine-tuning strategy to directly
model the distribution of target domain. Some works either impose strong
regularization [15, 39] or modify the network parameters with a slight
perturbation [30, 22, 25] to avoid overfitting to the limited target samples.
In addition, some data augmentation methods [28, 40, 41] are proposed to
enlarge the amount of the training data so as to improve the robustness of
generative models. However, these methods are only suitable for scenarios with
more than 100 training images. When the number of training images is reduced
to just a few (less than 10), the generative model usually generates images
with poor quality and suffers from early collapse.

Recently, as the pioneer work, intuited by the contrastive learning, Ojha _et
al_. [23] proposed to preserve the relative similarities and differences
between instances in the source domain via an instance distance consistency
loss (IDC for short). Given only 10 images, this method can generate more
diverse and realistic images for the target domain. Although IDC has made
great strides, the generated images still undergo identity degradation and
unnatural distortions or textures. The main reason is that IDC can not
guarantee the inherent structure of each image, leading to the drift of the
samples in the space of target domain (see Sec. 3.2 for a detailed
discussion).

In this work, we propose a relaxed spatial structural alignment (RSSA) method
to cope with the few shot generative model adaption task. It leverages richer
spatial structure priors of images from source domain to address the identity
degradation problem of the generative model. Specifically, we design a cross-
domain spatial structural consistency loss, which consists of self-correlation
consistency loss and disturbance correlation consistency loss. The former
helps align the self-correlation information of feature maps of the synthesis
pairs generated by the source and target generators, so as to constrain the
cross-domain consistency of inherent structural information. The latter helps
align the spatial mutual correlation between samples adjacent to each other in
the latent space, in order to constrain the cross-domain consistency of
variation tendency of a specific instance.

With the help of the cross-domain spatial structural consistency loss, the
samples from the target generator maintain original self-correlation and
disturbance correlation properties inherited from the source domain during
adaption. However, straightforward alignment may result in the dominant of the
attributes from the source domain in the optimization phase, and slow down the
model convergence. Thus, we propose to compress the latent space into a
subspace which is close to the target domain. This can relax the above
alignment because synthesis pairs generated from the subspace get closer to
each other.

To better evaluate the few shot generative model adaption methods, besides the
traditional quantitative metric and qualitative visualization, we design a
structural consistency score (SCS) which measures the structural similarities
of synthesis pairs from the source and target domains. Moreover, compared with
Inception Score (IS [27]) or Fréchet Inception Distance (FID [8]), SCS can
better reflect image identity preservation in few shot adaption.

The main contributions are summarized as follows:

  * •

We propose RSSA, a relaxed spatial structural alignment method, to transfer
rich spatial structural information of the large-scale source domain to the
few shot target domain with better identity preservation.

  * •

We introduce the latent space compression to relax the cross-domain alignment
via pulling synthesis pairs generated from the compressed subspace closer to
each other, and accelerate the training procedure.

  * •

We design a metric to evaluate the quality of synthesis images from the
structural perspective, which can serve as an alternative supplement to the
current metrics. Qualitatively and quantitatively, our method outperforms
existing competitors in a variety of settings.

##  2 Related Work

###  2.1 Few shot image generation

Few-shot image generation aims to generate diversified and high-quality images
in a new domain with a small amount of training data. The most straightforward
approach is to fine-tune a pre-trained GAN [32, 2, 16, 4]. However, fine-
tuning the entire network weights often leads to poor results. Researchers
proposed to modify part of the network weights [21, 25] or batch statistics
[22], and besides leverage different forms of regularization [15, 39] to avoid
overfitting. Wang _et al_. [30] introduced a miner network to steer the
sampling of the latent distribution to the target distribution. Data
augmentation strategies [28, 41, 40] were introduced to enlarge the amount of
the training data to improve the robustness of the generative model. However,
most of them fail in the extremely few shot setting (less than 10 images).
Recently, Ojha _et al_. [23] proposed to preserve the relative similarities
and differences between instances in the source domain via an instance
distance consistency loss. Different from work [23], we explore align the
distributions of the source and target domains from the perspective of spatial
structural consistency, and solve the problems of identity degradation and
image distortion during model adaption.

###  2.2 Image to image translation

The goal of the image to image translation is to convert an input image from a
source domain to a target domain with the intrinsic source content preserved
and the extrinsic target style transferred [24]. Variational autoencoders
(VAEs) and GANs are most commonly used and efficient deep generative models in
the image-to-image translation tasks [12, 20, 9, 17, 6, 19]. However, most
methods require a large amount of training data for both source and target
domains. Furthermore, generally they are not suitable for the few-shot
scenarios. Recent works [18, 31, 26] have begun to address this issue via
learning to separate the content and style factors, but require a large amount
of labeled data (class or style labels). Different from image to image
translation, we explore model-level adaption to the few shot target domain
rather than image-level translation. In addition, we do not rely on additional
labeled data.

Figure 2: The framework of our method. (a) an overview. $r$ is a small
disturbance corresponding to $z$. $P$ is a projector of compressing the latent
space as shown in (b). $L_{G_{s}\leftrightarrow G_{t}}$ is a cross-domain
spatial structural consistency loss, consisting of self-correlation
consistency and disturbance correlation consistency loss, as shown in (c).

##  3 Method

In this section, we first overview the definition of few shot generative model
adaption, and propose the framework of our method. Then, the two key
components of RSSA, cross-domain spatial structural consistency loss and
latent space compression are interpreted in detail in Sec. 3.2 and 3.3, the
optimization strategy is described in Sec. 3.4. Finally, a novel metric of
structural consistency score is proposed to better evaluate few shot
generation methods in Sec. 3.5.

###  3.1 Overview and framework

We have a generator $G_{s}$ pre-trained on a large-scale dataset in the source
domain $\mathcal{D}_{s}$. It can be considered as a function that maps a noise
vector $z$ sampled from the $d$-dimentional latent space $z\sim
p(z)\subset\mathbb{R}^{d}$ to a generated image $G_{s}(z)$ in the pixel space.
The goal of few shot adaption is to adapt $G_{s}$ from the source domain to
the target domain and obtain $G_{t}$, using a few samples in the target
domain. A standard fine-tuning approach is done by initializing $G_{t}$ with
$G_{s}$ and fine-tuning $G_{t}$ on the dataset in the target domain
$\mathcal{D}_{t}$ with an adversarial training procedure. The optimization
objectives are as below:

$L_{G}=-{\mathbb{E}}_{z\sim p(z)}[\log(D(G_{t}(z)))]$ (1) $\small
L_{D}={\mathbb{E}}_{x\sim\mathcal{D}_{t}}[\log(1-D(x)]+{\mathbb{E}}_{z\sim
p(z)}[\log(D(G_{t}(z)))],\small$ (2)

where $D$ represents a learnable discriminator.

Most of fine-tuning methods are easy to overfit in the extremely few shot
setting, because the discriminator can memorize the few examples and force the
generator to reproduce them. To solve this problem, we consider two
strategies. One is preserving the useful structure priors of images from the
source domain to restrict the generated images, so as to avoid identity
degradation during adaption. The other is compressing the latent space to a
subspace where synthesis images of the source and target domains are pulled
closer to each other to relax the structural constraint. Fig. 2(a) shows our
pipeline. During the model adaption, we conduct relaxed spatial structural
alignment. A cross-domain spatial structural consistency loss
$L_{G_{s}\leftrightarrow G_{t}}$ (Fig. 2(c)) and a projector $P$ of latent
space compression (Fig. 2(b)) are introduced.

###  3.2 Cross-domain spatial structural consistency loss

IDC [23] preserves the relative distances between instances in the source
domain and achieves the state-of-the-art (SOTA) performance in few shot
setting. However, the structure of generated images distorts, leading to the
problem of identity degradation. Abundant evidences can be found in Fig. 5, 6
and 7. The main reason is that IDC can not guarantee the inherent structure of
each image, leading to the drift of the generated samples in the target domain
space. As shown in Fig. 3(a), the generated images of the target domain (the
green points) maintain the correct instance distances, but deviate from their
correct positions (the red points). To avoid this drift, we propose a cross-
domain spatial structural consistency loss, which preserves the inherent
spatial structure and the variation tendency of images from the source domain
as shown in Fig. 3(b). Specifically, we design a self-correlation consistency
loss to constrain the inherent structure of the images, and a disturbance
correlation consistency loss to constrain the variation tendency under a
certain disturbance of the images.

Self-correlation consistency loss. We are inspired by some relevancy mining
methods[19, 13, 14] and adopt the self-correlation matrices of feature maps at
each each convolutional layer to formulate the inherent structure information
of the image. Each pair of self-correlation matrices from the same layers of
$G_{s}$ and $G_{t}$ are constrained with the $\text{smooth-}{\ell}_{1}$ loss
[7], so as to ensure that the images in source and target domains have similar
inherent structure. Define $f^{l}\in\mathbb{R}^{c\times w\times h}$ as the
feature maps at the $l^{th}$ layer. $f^{l}(x,y)$ is a $c$ dimensional vector.
Each entry of the self-correlation matrix $C^{l}_{x,y}\in\mathbb{R}^{w\times
h}$ of the position $(x,y)$ at the $l^{th}$ layer can be calculated as below:

$C^{l}_{x,y}(i,j)=\text{cos}(f^{l}(x,y),f^{l}(i,j)),$ (3)

where cos($\cdot$) denotes the cosine similarity function, $(i,j)$ is the
corresponding position in $f^{l}$. The spatial self-correlation consistency
loss between $G_{s}$ and $G_{t}$ can be calculated as

$\small L_{\text{scc}}(G_{t},G_{s})={\mathbb{E}}_{z_{i}\sim
p(z)}\sum_{l}\sum_{x,y}\text{smooth-}{\ell}_{1}(C_{x,y}^{t,l},C_{x,y}^{s,l}),$
(4)

where $C_{x,y}^{t,l}$ and $C_{x,y}^{s,l}$ indicate the self-correlation
matrices of the position $(x,y)$ at the $l^{th}$ layer for $G_{t}$ and
$G_{s}$.

Note that computation of the self-correlation matrices is an $O((w\cdot
h)^{2})$ operation. For feature maps with high resolution, we first aggregate
the adjacent feature vectors by adopting average pooling and break the whole
feature map into patches to compute local self-correlation matrices.

Figure 3: Illustration of the spatial structural alignment. (a) instance
distance consistency loss from IDC [23]. (b) our cross-domain spatial
structural consistency loss.

Disturbance correlation consistency loss. The latent space of a generative
model is continuous rather than discrete, hence we propose to model the
variation tendency under certain disturbances of the images(essentially the
gradient information around each instance). Specifically, we take an input
noise vector as an anchor point, and then sample a batch of vectors from a
small neighborhood of this anchor point. The spatial similarities between
these samples are calculated and transferred from the source domain to the
target domain.

For an input noise $z_{i}$, define a neighborhood with radius $r$,
$U(z_{i},r)=\\{z||z-z_{i}|<r\\}$. We sample $N$ noise vectors from
$U(z_{i},r)$ and form a batch of $N+1$ vectors
$\left\\{z_{n}\right\\}_{1}^{N+1}$ to represent the neighborhood. Define
$D^{l}_{jk}$ as the pixel-wise spatial mutual correlation for the $l^{th}$
layer feature map $f^{l}$ between any two samples $z_{j}$ and $z_{k}$ from
$\left\\{z_{n}\right\\}_{1}^{N+1}$. $D^{l}_{jk}$ at position $(x,y)$ is
denoted by the softmax of similarities between feature vector at $(x,y)$ in
$f^{l}_{j}$ and a small corresponding region
$Q=\\{(m,n)|x-\frac{\delta}{2}<m<x+\frac{\delta}{2},y-\frac{\delta}{2}<n<y+\frac{\delta}{2}\\}$,
where $\delta$ is the width of a slide window, in $f^{l}_{k}$ as below:

$\small
D^{l}_{jk}(x,y)=\text{Softmax}(\\{\text{cos}(f^{l}_{j}(x,y),f^{l}_{k}(m,n))\\}_{(m,n)\in
Q}),\small$ (5)

where cos($\cdot$) denotes the cosine similarity function.

On basis of the computed pixel-wise correlation distribution, we impose the
disturbance correlation consistency constraint by minimizing the L1 distance:

$\footnotesize L_{\text{dcc}}(G_{t},G_{s})={\mathbb{E}}_{z_{i}\sim
p(z)}\sum_{l,j,k,x,y}\left\|D^{t,l}_{jk}(x,y)-D^{s,l}_{jk}(x,y)\right\|_{1},\footnotesize$
(6)

The spatial structural consistency loss $L_{G_{s}\leftrightarrow G_{t}}$ is
composed of the self-correlation consistency loss $L_{\text{scc}}$ and
disturbance correlation consistency loss $L_{\text{dcc}}$. It is calculated as
below:

$L_{G_{s}\leftrightarrow G_{t}}=\alpha L_{\text{scc}}+\beta L_{\text{dcc}}.$
(7)

where $\alpha$, $\beta$ are ratio parameters.

###  3.3 Latent space compression

The spatial structural consistency loss helps align the generated images of
the source and target domains. However, straightforward alignment may cause
the dominant of the attributes from the source domain, and thus slow down
model adaption. Therefore, we propose to compress the latent space to a
subspace close to the target domain to relax the cross-domain alignment.
Specifically, we first invert the few samples from target domain
$\\{x^{t}_{i}\\}_{i=1}^{n}$ to the $W^{+}$ space of $G_{s}$ by utilizing
Image2StyleGAN [1]. Given $n$ target samples, an inverted latent code set at
$l^{th}$ layer is denoted as $\\{w^{l}_{i}\\}_{i=1}^{n}$. Define a $n$ column
matrix $A^{l}$ which is composed of $\\{w^{l}_{i}\\}_{i=1}^{n}$ by
$A^{l}_{*i}=w^{l}_{i}$, and hence we obtain a subspace $\mathcal{X}^{l}$ of
the $l^{th}$ latent space, where $\mathcal{X}^{l}$ is equivalent to the column
space of $A^{l}$. Given an input noise $z_{j}$, the corresponding latent code
in the $l^{th}$ layer is $w_{j}^{l}$, the corresponding modulation coefficient
is $\alpha^{l}$, we modulate $w_{j}^{l}$ and project it onto the $l^{th}$ sub-
plane $\mathcal{X}^{l}$ via least square method:

$\displaystyle\overline{w_{j}^{l}}$
$\displaystyle=A^{l}({A^{l}}^{\top}A^{l})^{-1}{A^{l}}^{\top}w_{j}^{l}$ (8)
$\displaystyle\hat{w_{j}^{l}}$
$\displaystyle=\alpha^{l}\overline{w_{j}^{l}}\frac{\left\|w_{j}^{l}\right\|}{\left\|\overline{w_{j}^{l}}\right\|}+(1-\alpha^{l})w_{j}^{l},$

where $\hat{w_{j}^{l}}$ is the projected code at $l^{th}$ for $z_{j}$.

In this way, we compress the original latent space into a narrow subspace
close to the target domain. Images generated by $G_{s}$ with the latent codes
sampled from the compressed subspace imply some characteristics of the target
domain. As shown in Fig. 4, generated images (bottom row) shows some
characteristics of sketch on the texture and color. By sampling latent codes
from the compressed subspace before the alignment, we are capable to stabilize
and accelerate the whole training procedure. Note that latent codes of
different layers modulate the output images at distinct semantic levels in
StyleGAN [10] architecture. We set large $\alpha_{i}$ at the top layers and
small ones at the bottom layers to alter the generated images’ attributes at
high semantic levels while maintaining the original spatial structure to a
great extent.

Figure 4: Generated images of $G_{s}$ with input latent codes sampled from
different spaces. Top: original latent space. Bottom: compressed subspace.
Setting: Flickr-Faces $\rightarrow$ Sketches.

###  3.4 Optimization

We follow an adversarial optimization procedure. The objective of $G_{t}$ is a
combination of $L_{G}$ (Eq.1) and $L_{G_{s}\leftrightarrow G_{t}}$ (Eq.7). We
simply set the hyper-parameters $\alpha$ and $\beta$ as 1 for all experiments.
The objective of $D$ is the same with [23] by utilizing a combination of
image-wise and patch-wise discriminant loss. Standard path regularization loss
to $G_{t}$ and gradient penalization loss to $D$ are also adopted at every
several iterations.

###  3.5 Evaluation metric

In addition to the intuitively visual evaluation, IS [27] and FID [8] are the
most widely used quantitative evaluation metrics for image generative models.
However, both of them map the generated images to the feature space by an
Inception network, which can not quantify the quality of the spatial structure
of the generated images. Meanwhile, in few shot setting, many fine-tuning
based methods are inclined to simply synthesis images similar with the
training samples given arbitrary input noises. Yet, this may obtain high IS in
some cases which is counter-intuitive, see Table 1. Furthermore, computing FID
requires a large number of realistic images of the target domain, which is
impractical in the few shot setting. Therefore, we adopt IS as a general
evaluation metric and propose a novel spatial structure evaluation metric,
termed structural consistency score, to cover the shortage of IS.

Structural Consistency Score (SCS). For a image pair $\left\langle
x^{s},x^{t}\right\rangle$ generated by $\left\langle G_{s},G_{t}\right\rangle$
with the same input $z_{i}$, we claim that $x^{t}$ preserves structural
consistency when it can be easily recognized as a derivative sample from
$x^{s}$. Inspired by [36], we extract a meaningful edge map of one image to
represent its structural information by HED[33]. The SCS of a generated image
of the target domain $x^{t}$ is computed with the dice similarity coefficient
[5] between the edge maps of $x^{t}$ and $x^{s}$. The formalization is as
below:

$\displaystyle\text{SCS}(x^{t})$ $\displaystyle=\frac{2|H(x^{t})\cap
H(x^{s})|}{|H(x^{t})|+|H(x^{s})|},$ (9)

where $H(\cdot)$ denotes HED function. $|H(x^{t})\cap H(x^{s})|$ is calculated
by the pixel-wise inner product of $H(x^{t})$ and $H(x^{t})$. $|H(x^{t})|$ and
$|H(x^{s})|$ is calculated by the sum of squares of matrix elements. Then, the
SCS of the target GAN $G_{t}$ can be quantified into the expectation as below:

$\displaystyle\text{SCS}(G_{t})=\mathbb{E}_{z_{i}\sim
p(z)}[\frac{2|H(G_{t}(z_{i}))\cap
H(G_{s}(z_{i}))|}{|H(G_{t}(z_{i}))|+|H(G_{s}(z_{i}))|}].$ (10)

Higher SCS means better spatial structural consistency between $G_{t}$ and
$G_{s}$. It is remarkable that the SCS of an overfitted model will be very
low, because it can not generate images with structures similar to the source
domain images.

##  4 Experiments

In this section, we demonstrate the effectiveness of RSSA in few shot setting.
Qualitative and quantitative comparisons between our method and several
baselines, TGAN [32], FreezeD [21], MineGAN [30], IDC [23]. As the SOTA
method, IDC [23] is our primary comparison method in most experiments.

We adopt the StyleGANv2 [11] pre-trained on three different datasets: (i)
Flickr-Faces-HQ (FFHQ) [10], (ii) LSUN Churches [37], (iii) LSUN Cars [37]. We
adapt the source GANs to various target domains including: (i) face sketches
[29], (ii) face paintings by Van Gogh [34], (iii) face paintings by Moise
Kisling [34], (iv) haunted houses [23], (v) village painting by Van Gogh [23],
(vi) wrecked/abandoned cars [23]. Model adaptions are done in 10-shot, 5-shot
and 1-shot settings.

Figure 5: Comparison results with different methods on Flickr-Faces
$\rightarrow$ Sketches (10-shot). Figure 6: Comparison results between IDC and
our method on Flickr-Faces $\rightarrow$ Vincent van Gogh (left), Moise
Kisling (right) (10-shot). Figure 7: Comparison results between IDC and our
method on Churches $\rightarrow$ Van Gogh Village (left), Haunted Houses
(right) (5-shot).

Metric | Method | f$\rightarrow$S | f$\rightarrow$V | f$\rightarrow$M | c$\rightarrow$V | c$\rightarrow$H
---|---|---|---|---|---|---
10-shot | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot
IS | TGAN | 2.00±0.04 | 2.09±0.05 | 2.69±0.18 | 1.65±0.07 | 1.88±0.08 | 1.98±0.04 | 2.82±0.09 | 2.28±0.05 | 5.16±0.19 | 4.47±0.14
FreezeD | 1.98±0.05 | 2.05±0.06 | 2.46±0.06 | 1.77±0.03 | 1.91±0.06 | 1.85±0.04 | 2.78±0.07 | 2.32±0.09 | 5.29±0.15 | 4.60±0.17
MineGAN | 1.92±0.03 | 2.12±0.07 | 2.52±0.09 | 1.71±0.04 | 1.84±0.03 | 1.87±0.08 | 2.51±0.05 | 2.18±0.08 | 5.22±0.11 | 4.55±0.18
IDC | 1.95±0.02 | 2.15±0.04 | 1.55±0.04 | 1.52±0.02 | 1.97±0.03 | 1.76±0.06 | 2.86±0.11 | 2.78±0.06 | 5.44±0.21 | 5.15±0.13
Ours | 2.10±0.03 | 2.41±0.03 | 1.77±0.06 | 1.61±0.04 | 2.17±0.07 | 2.09±0.05 | 3.54±0.10 | 3.62±0.13 | 6.26±0.18 | 5.70±0.20
SCS | TGAN | 0.287 | 0.285 | 0.380 | 0.375 | 0.344 | 0.350 | 0.355 | 0.343 | 0.211 | 0.218
FreezeD | 0.289 | 0.288 | 0.384 | 0.372 | 0.347 | 0.346 | 0.353 | 0.349 | 0.216 | 0.212
MineGAN | 0.294 | 0.290 | 0.340 | 0.332 | 0.339 | 0.341 | 0.375 | 0.350 | 0.211 | 0.214
IDC | 0.437 | 0.422 | 0.594 | 0.568 | 0.524 | 0.494 | 0.551 | 0.535 | 0.490 | 0.424
Ours | 0.511 | 0.507 | 0.702 | 0.685 | 0.644 | 0.618 | 0.702 | 0.681 | 0.573 | 0.582

Table 1: Quantitative evaluation of methods by IS and SCS. f and c represent
faces and churches source domains. S,V,M,H represent four target domains:
sketches, Van Gogh’s paintings, Moise Kisling’s paintings and haunted houses.
Best results are bold.

###  4.1 Performance evaluation

Qualitative comparison.

Figure 8: Results of our method on Cars $\rightarrow$ Wrecked Cars, Faces
$\rightarrow$ Vincent van Gogh, and Churches $\rightarrow$ Haunted houses
(1-shot).

Fig. 5 shows results on FFHQ $\rightarrow$ sketches using different adaption
methods. We can observe that TGAN overfits strongly to the samples of the
target domain. Compared with TGAN, FreezeD and MineGAN do not improve the
results. This indicates that although they could play a positive role in a
small dataset with more than 100 training samples, they would be ineffective
in extremely few shot setting (less than 10). By contrast, IDC improves the
correspondence between the source domain and the target domain, and shows
similar visual patterns between the synthesis pairs. Further, as for images
synthesised by RSSA, one can easily recognize the corresponding source domain
images with only few glances. This is because our method acquires visual
attributes from the target domain and meanwhile greatly preserves the spatial
structural information of images from the source domain.

In order to comprehensively compare IDC with our method, we extend comparison
experiments to multiple target domains with different few shot setting as
shown in Fig. 6 (10-shot) and Fig. 7 (5-shot). We can observe the distorted
attributes of human faces (Fig. 6) and texture degradation of churches (Fig.
7) in IDC’s results, but there are almost no similar phenomenons in our
results. In addition, we take a bold stab at 1-shot scenarios as shown in Fig.
8, and obtain some decent results. Good visual results are mainly attributed
to the cooperation of latent space compression and spatial structure
alignment, the former helps acquire attributes from the target domain faster,
the latter helps preserve the structural knowledge from the source domain.

Quantitative comparison. To quantify the quality and diversity of the
synthesis images, we evaluate all methods with IS and SCS. All quantitative
experiments are conducted in 10-shot and 5-shot settings. For IS, we calculate
means and variances over 10 runs on 10000 randomly sampled images. As shown in
Table 1, our method achieves best scores in most cases due to the high
diversity and quality of synthesis images. However, TGAN, FreezeD and MineGAN
outperforms IDC and RSSA on face$\rightarrow$Van Gogh’s paintings. The reason
is that they simply overfit to the few training samples, while images
generated by IDC and RSSA tend to remix the textures and colors of the
paintings. This indicates that IS sometimes fails to handle the overfitting
problem in few shot setting. For SCS, we randomly sample 500 noise vectors as
inputs of $G_{s}$ and $G_{t}$, then form the synthesis pairs and calculate
their mean score. As shown in Table 1, TGAN, FreezeD and MineGAN overfit to
the training samples and obtain lower results for all settings. IDC performs
much better by preserving the distances of instances. RSSA consistently
surpasses all the comparison methods by a large margin due to the better
spatial structure preservation. To visually understand SCS, Fig. 9 shows two
groups of edge extraction examples of on FFHQ $\rightarrow$ sketches and
Churches $\rightarrow$ Van Gogh Village. Obviously, our generated images
retain more accurate edge information than those from IDC, thus obtain higher
SCS.

Figure 9: Comparison of edge maps between IDC and our method. Top row shows
generated images, bottom raw shows corresponding edge maps. Left: IDC. Middle:
Source. Right: Ours

###  4.2 Ablation study

Effect of the spatial structural consistency loss. We conduct ablation
experiments to verify the effectiveness of the two components of our proposed
spatial structural consistency loss. As shown in Fig. 10, $L_{scc}$ and
$L_{dcc}$ greatly improve the visual quality of synthesis images separately
and get the best visual results in cooperation. Consistently, quantitative
conclusion can be drawn in Table 2. The reason is that they introduce inherent
and neighbouring structural constraint separately and share good
compatibility.

Figure 10: Qualitative ablation of the loss function on FFHQ $\rightarrow$ sketches in 10-shot setting. Method | IS | SCS
---|---|---
f$\rightarrow$V |  c$\rightarrow$H |  f$\rightarrow$V |  c$\rightarrow$H
without $L_{G_{s}\leftrightarrow G_{t}}$ |  1.40±0.03 |  5.38±0.09 | 0.449 | 0.369
with $L_{\text{scc}}$ |  1.69±0.05 |  6.18±0.17 | 0.684 | 0.544
with $L_{\text{dcc}}$ |  1.64±0.05 |  6.13±0.14 | 0.671 | 0.548
with $L_{G_{s}\leftrightarrow G_{t}}$ | 1.77±0.06 | 6.26±0.18 | 0.702 | 0.573
Table 2: Quantitative ablation of the loss function in 10-shot setting. f and
c represent faces and churches source domains, V and H represent Van Gogh’s
paintings and haunted houses target domains.

Effect of the latent space compression. We conduct the model adaption on FFHQ
$\rightarrow$ sketches in 10-shot setting. As shown in Fig. 11, we observe
that some target-domain irrelevant attributes (e.g. background and color)
degrade faster when adopting latent space compression. This demonstrates that
(1) the latent space compression copes with the dominant of attributes from
source domain well so as to relax the spatial structural alignment; (2) it
helps speed up the adaption procedure of $G_{t}$.

Figure 11: Ablation of the latent space compression, which shows the output
images from the same input noises at different training iterations. $P$
denotes the projector of compression.

##  5 Conclusion and Limitations

In this paper, we propose a novel few shot generative model adaption method,
relaxed spatial structural alignment (RSSA). By aligning the generative
distributions of the source and target domains via a cross-domain spatial
structural consistency loss, the inherent structure information and spatial
variation tendency of images from the source domain can be well preserved and
transferred to the target domain. The original latent space is compressed to a
narrow subspace close to the target domain, which relaxes the cross-domain
alignment and accelerates the convergence rate of the target domain generator.
In addition, we design a novel metric, SCS, to assess the structural quality
of generated images. It may serve as an alternative supplement to the current
metrics in the few shot generation scenarios.

Although our method can deal with the settings of extremely few shot training
samples well and generate compelling visual results, it also has some
limitations. The spatial structural consistency loss is not friendly to some
abstract domains, such as paintings from Amedeo Modigliani, who is known for
portraits in a modern style characterized by a surreal elongation of faces.
Nevertheless, we believe that more data-efficient generative models will be
proposed in the near future. The application of these models will in turn
facilitate a wide range of downstream tasks such as few shot image
classification.

Acknowledgement. This work was supported in part by the National Key R&D
Program of China under Grant 2018AAA0102000, and in part by the National
Natural Science Foundation of China: U21B2038, U19B2038, 61931008, 61732007,
and CAAI-Huawei MindSpore Open Fund, Youth Innovation Promotion Association of
CAS under Grant 2020108, CCF-Baidu Open Fund.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In ICCV, pages 4432–4441, 2019.
  * [2] Sergey Bartunov and Dmitry Vetrov.  Few-shot generative modelling with generative matching networks.  In AISTATS, pages 670–678. PMLR, 2018.
  * [3] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  In ICLR, 2018.
  * [4] Louis Clouâtre and Marc Demers.  Figr: Few-shot image generation with reptile.  arXiv preprint arXiv:1901.02199, 2019.
  * [5] Lee R Dice.  Measures of the amount of ecologic association between species.  Ecology, 26(3):297–302, 1945.
  * [6] Yuke Fang, Weihong Deng, Junping Du, and Jiani Hu.  Identity-aware cyclegan for face photo-sketch synthesis and recognition.  Pattern Recognition, 102:107249, 2020.
  * [7] Ross Girshick.  Fast r-cnn.  In ICCV, pages 1440–1448, 2015.
  * [8] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  NeurIPS, 30, 2017.
  * [9] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In CVPR, pages 1125–1134, 2017.
  * [10] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, pages 4401–4410, 2019.
  * [11] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In CVPR, pages 8110–8119, 2020.
  * [12] Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang.  Diverse image-to-image translation via disentangled representations.  In ECCV, pages 35–51, 2018.
  * [13] Liang Li, Xingyu Gao, Jincan Deng, Yunbin Tu, Zheng-Jun Zha, and Qingming Huang.  Long short-term relation transformer with global gating for video captioning.  TIP, 2022.
  * [14] Liang Li, Chenggang Clarence Yan, Xing Chen, Chunjie Zhang, Jian Yin, Baochen Jiang, and Qingming Huang.  Distributed image understanding with semantic dictionary and semantic expansion.  Neurocomputing, 174:384–392, 2016.
  * [15] Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman.  Few-shot image generation with elastic weight consolidation.  arXiv preprint arXiv:2012.02780, 2020.
  * [16] Weixin Liang, Zixuan Liu, and Can Liu.  Dawson: A domain adaptive few shot generation framework.  arXiv preprint arXiv:2001.00576, 2020.
  * [17] Jianxin Lin, Yingxue Pang, Yingce Xia, Zhibo Chen, and Jiebo Luo.  Tuigan: Learning versatile image-to-image translation with two unpaired images.  In ECCV, pages 18–35. Springer, 2020.
  * [18] Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz.  Few-shot unsupervised image-to-image translation.  In ICCV, pages 10551–10560, 2019.
  * [19] Zhenahuan Liu, Liang Li, Huajie Jiang, Xin Jin, Dandan Tu, Shuhui Wang, and Zheng-Jun Zha.  Unsupervised coherent video cartoonization with perceptual motion consistency.  In AAAI, 2022.
  * [20] Liqian Ma, Xu Jia, Stamatios Georgoulis, Tinne Tuytelaars, and Luc Van Gool.  Exemplar guided unsupervised image-to-image translation with semantic consistency.  In ICLR, 2018.
  * [21] Sangwoo Mo, Minsu Cho, and Jinwoo Shin.  Freeze the discriminator: a simple baseline for fine-tuning gans.  arXiv preprint arXiv:2002.10964, 2020.
  * [22] Atsuhiro Noguchi and Tatsuya Harada.  Image generation from small datasets via batch statistics adaptation.  In ICCV, pages 2750–2758, 2019.
  * [23] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot image generation via cross-domain correspondence.  In CVPR, pages 10743–10752, 2021.
  * [24] Yingxue Pang, Jianxin Lin, Tao Qin, and Zhibo Chen.  Image-to-image translation: Methods and applications.  arXiv preprint arXiv:2101.08629, 2021.
  * [25] Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang.  Few-shot adaptation of generative adversarial networks.  arXiv preprint arXiv:2010.11943, 2020.
  * [26] Kuniaki Saito, Kate Saenko, and Ming-Yu Liu.  Coco-funit: Few-shot unsupervised image translation with a content conditioned style encoder.  In ECCV, pages 382–398. Springer, 2020.
  * [27] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  NeurIPS, 29:2234–2242, 2016.
  * [28] Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Nguyen, and Ngai-Man Cheung.  On data augmentation for gan training.  TIP, 30:1882–1897, 2021.
  * [29] Xiaogang Wang and Xiaoou Tang.  Face photo-sketch synthesis and recognition.  PAMI, 31(11):1955–1967, 2008.
  * [30] Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer.  Minegan: effective knowledge transfer from gans to target domains with few images.  In CVPR, pages 9332–9341, 2020.
  * [31] Yaxing Wang, Salman Khan, Abel Gonzalez-Garcia, Joost van de Weijer, and Fahad Shahbaz Khan.  Semi-supervised learning for few-shot image-to-image translation.  In CVPR, pages 4453–4462, 2020.
  * [32] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu.  Transferring gans: generating images from limited data.  In ECCV, pages 218–234, 2018.
  * [33] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  In ICCV, pages 1395–1403, 2015.
  * [34] Jordan Yaniv, Yael Newman, and Ariel Shamir.  The face of art: landmark detection and geometric style in portraits.  TOG, 38(4):1–15, 2019.
  * [35] Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do.  Semantic image inpainting with deep generative models.  In CVPR, pages 5485–5493, 2017.
  * [36] Ran Yi, Yong-Jin Liu, Yu-Kun Lai, and Paul L Rosin.  Unpaired portrait drawing generation via asymmetric cycle mapping.  In CVPR, pages 8217–8225, 2020.
  * [37] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  arXiv preprint arXiv:1506.03365, 2015.
  * [38] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In CVPR, pages 5505–5514, 2018.
  * [39] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee.  Consistency regularization for generative adversarial networks.  In ICLR, 2019.
  * [40] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.  Differentiable augmentation for data-efficient gan training.  In NeurIPS, 2020.
  * [41] Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang.  Image augmentations for gan training.  arXiv preprint arXiv:2006.02595, 2020.
  * [42] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In ICCV, pages 2223–2232, 2017.


# Multimodality Multi-Lead ECG Arrhythmia Classification using Self-Supervised
Learning

1st Thinh Phan  Department of AI Convergence
Chonnam National University
Gwangju, South Korea 61186
<EMAIL_ADDRESS>2nd Duc Le  Department of CSCE
University of Arkansas,
Fayetteville, Arkansas 72703
<EMAIL_ADDRESS>3rd Patel Brijesh  Department of Cardiology
West Virginia University
Morgantown, WV 26506
<EMAIL_ADDRESS>4th Donald Adjeroh  Department of CSEE
West Virginia University
Morgantown, WV 26506
<EMAIL_ADDRESS>5th Jingxian Wu  Department of ELEG
University of Arkansas,
Fayetteville, Arkansas 72703
<EMAIL_ADDRESS>6th Morten Olgaard Jensen  Department of Biomedical Engineering
University of Arkansas,
Fayetteville, Arkansas 72703
<EMAIL_ADDRESS>7th Ngan Le  Department of CSCE
University of Arkansas,
Fayetteville, Arkansas 72703
<EMAIL_ADDRESS>
###### Abstract

Electrocardiogram (ECG) signal is one of the most effective sources of
information mainly employed for the diagnosis and prediction of cardiovascular
diseases (CVDs) connected with the abnormalities in heart rhythm. Clearly,
single modality ECG (i.e. time series) cannot convey its complete
characteristics, thus, exploiting both time and time-frequency modalities in
the form of time-series data and spectrogram is needed. Leveraging the
cutting-edge self-supervised learning (SSL) technique on unlabeled data, we
propose SSL-based multimodality ECG classification. Our proposed network
follows SSL learning paradigm and consists of two modules corresponding to
pre-stream task, and down-stream task, respectively. In the SSL-pre-stream
task, we utilize self-knowledge distillation (KD) techniques with no labeled
data, on various transformations and in both time and frequency domains. In
the down-stream task, which is trained on labeled data, we propose a gate
fusion mechanism to fuse information from multimodality.To evaluate the
effectiveness of our approach, ten-fold cross validation on the 12-lead
PhysioNet 2020 dataset has been conducted. https://github.com/UARK-
AICV/ECG_SSL_12Lead.

###### Index Terms:

ECG classification, self-supervised learning, contrastive learning,
multimodalities, multi-lead

##  I Introduction

CVDs are leading causes of deaths globally. The mortality rate can be
considerably reduced by early treatment if occult signals linked with CVD are
detected by ECG. The ECG signals, which record cardiac electrical activities,
are widely adopted to diagnose abnormal heart rhythms and intra-cardiac
conduction abnormalities.

Traditionally, cardiac feature extraction [1] and pattern classifiers [2] are
separated. Notwithstanding the decent performance, they are not applicable to
real-life cases because of high time consumption and complexity. Recent DNNs-
based approaches have obtained amazing research progress in various domains
[3, 4, 5]. DNN-based ECG classification in general can be categorized into
single lead classification [6, 7] or multiple lead classification [8, 9]. _Our
proposed method belongs to the second category_. In this group, [8] proposed a
simple residual neural network used for classifying 6 types of abnormalities
on the in-house 12-lead database, where some of its testing metrics are better
than those of expert cardiologists. [9] trained a CNNs-based structure on
multi-lead ECG data to diagnose myocardial infraction. Due to high complexity
in higher-dimensional data of 12-lead ECG, different architectures have been
adopted to model time correlation among ECG sample points [10], [11]. Besides
time-series data, time-frequency has played an important role in ECG analysis.
[12] used STFT-based spectrogram and 2D CNNs for ECG arrhythmia
classification. [7] proposed using STFT and stationary wavelet transform (SWT)
transformations to obtain two-dimensional (2-D) matrix input suitable for deep
CNNs. [13] proposed a novel wavelet sequence based on deep bidirectional LSTM
network model.

Furthermore, with the growing demand for medical examination and treatment,
the healthcare industry steadily accumulates innumerable amounts of data but
these unlabelled data might not be serviceable for most tasks. To address this
limitation, we leverage the recent advanced SSL techniques, i.e., contrastive
learning [14]. A common workflow to apply SSL is to train the network in an
unsupervised manner by learning with a pre-stream task, and then finetuning
the pre-trained network on a target downstream task. The suitable pre-stream
tasks can be considered in four categories: context-based [15], generation-
based [16], free semantic label-based [17, 18], and cross-modal-based [19,
20]. Recently, SSL-based DNNs have also been applied in ECG classification.
[21] proposed a self-designed loss to bring the representations from the same
patient closer and study the shared context of individual recordings through
time and scenarios. [22] customized an SSL model that understands the
differences among segments from individual patients and dissimilarities
between patients’ recordings from same category. [23] conducted extensive
experiments on four SSL methods with multiple combinations of transformations
and demonstrated the improvement in macro AUC when using SSL. However, the
existing SSL-based ECG classification approaches only focus on time domain
with time-series data and they have not exploited the ECG characteristics in
frequency domain.

In this work, we adopt the advanced SSL technique, i.e., self-distillation SSL
with no labels [14, 24]. Our proposed network contains two modules i.e., SSL
pre-stream task, SSL down-stream task. The first module is trained on
unlabeled data and contains two components corresponding to 1D CNNs for time-
series data and 2D CNNs for time-frequency spectral (spectrogram) signal. The
second module fine-tunes the pre-trained models i.e. 1D CNNs and 2D CNNs from
the first module to perform muti-lead ECG classification on labeled data. In
the down-stream task, the features from the two networks are fused by our
proposed gate fusion mechanism. The overall flowchart of our proposed network
is shown in Fig. 1.

Figure 1: Overall flowchart of our proposed network consisting two modules
i.e., SSL pre-stream task and SSL down-stream task. In the SSL pre-stream
task, the pre-trained models are student networks and they are trained on
unlabeled data with self-KD mechanism. In the SSL pre-stream task, the pre-
trained models are fine-tuned on labeled data. The features $f_{1}$ from time-
series and $f_{2}$ from spectrogram are fused by our proposed gate fusion.

##  II Proposed Methods

###  II-A Self-Knowledge Distillation (KD): a revisit

KD [25] is a learning paradigm where a student network $\mathcal{S}_{\theta}$
is trained to match the output of a given teacher network
$\mathcal{T}_{\phi}$, parameterized by $\theta$ and $\phi$, respectively.
Given an input signal $x$, self-KD is explained in the following steps: (i)
Apply different transformations ($\mathbb{T}_{1}$, $\mathbb{T}_{2}$) on $x$ to
generate global view $x_{1}=\mathbb{T}_{1}(x)$ and local view
$x_{2}=\mathbb{T}_{2}(x)$ at different distorted views, or crops. On the same
transformation, the global transformation $\mathbb{T}^{1}_{1}$ and local
transformation $\mathbb{T}^{1}_{2}$ are defined with different crop ratios
i.e. greater than 50% is considered as global; otherwise, it is local. ; (ii)
Pass $x_{1}$ and $x_{2}$ to networks of $\mathcal{S}_{\theta}$ and
$\mathcal{T}_{\phi}$; (iii) Compute similarity between probability
distributions of output from $\mathcal{S}_{\theta}$ and $\mathcal{T}_{\phi}$
using a cross-entropy loss. Notably, the output of $\mathcal{T}_{\phi}$ is
centered with a mean computed over the batch.

Unlike typical KD, Self-KD[14] build $\mathcal{T}_{\phi}$ using the past
iterations of $\mathcal{S}_{\theta}$. Thus, both $\mathcal{T}_{\phi}$ and
$\mathcal{S}_{\theta}$ are sharing the same network architecture with
different sets of parameters $\theta$ and $\phi$. Notably, both
$\mathbb{T}_{1}$ and $\mathbb{T}_{2}$ are sharing the same transformations
with different crop ratios. The overall flowchart of self-KD is illustrated in
Fig. 2.

Figure 2:  Illustration of SSL with self-KD mechanism with no labeled data.

###  II-B SSL: Pre-stream Task

The pre-stream task makes use of self-KD technique to pre-train models of
time-series data and time-frequency signal using two components as shown in
the yellow block in Fig.1. To apply self-KD technique to two different
modalities, we need to define the network architectures of
$\mathcal{T}_{\phi}$, $\mathcal{S}_{\theta}$ and transformations of
$\mathbb{T}_{1}$ $\mathbb{T}_{2}$ as follows:

1\. Pre-stream task with time-series data This component takes time-series
$\mathcal{X}$ as an input. The SSL KD in this task is performed by Time Cutout
(TC) and Gaussian Noise (GN) transformation $\mathbb{T}^{1}$ as follows:

Let $t_{1}$ and $t_{2}$ denote the start and the end of a random crop on
signal $\mathcal{X}$, TC transformation is defined as follows:

$\mathcal{X}_{i,TC}:\\{X_{i}\\}_{i={t}_{1}}^{{t}_{2}}=0$ (1)

where $t_{1}=rand(0,n-\alpha),t_{2}=t_{1}+\alpha$,
$\alpha=\mathcal{U}(0,0.5).n$, with $n$ denoting the length of $\mathcal{X}$,
and $\mathcal{U}$ denoting the uniform distribution.

Let $\mathcal{N}(0,\sigma)$ denote the Gaussian Noise (GN). The GN
transformation is defined as:
$\mathcal{X}_{i,GN}:\mathcal{X}_{i}+\mathcal{N}(0,\sigma)$ The
$\mathbb{T}^{1}$ transformation is
$\mathbb{T}^{1}:\mathcal{X}_{i,TC}+\mathcal{X}_{i,GN}$.

We utilize xresnet1d50 [26] as our backbone network for both
$\mathcal{T}^{1}_{\phi}$ and $\mathcal{S}^{1}_{\theta}$, which are trained on
different parameters of $\phi$ and $\theta$.

2\. Pre-stream task with time-frequency data This component takes time-
frequency $\mathcal{F}$ as an input. $\mathcal{F}$ is generated by applying
Short Time Fourier Transform (STFT) to the time-series $\mathcal{X}$. The
equation for STFT is shown in Eq:2 where $\mathcal{S}$ is the STFT function
and $g(n-m)$ is the window function. Usually a Hann or a Gaussian window is
used and the width of window is specified by $m$.

$\begin{split}\\{s_{i}\\}_{i=1}^{n}&=\mathcal{S}(\\{\mathcal{X}_{i}\\}_{i=1}^{n})\\\
\mathcal{S}(\\{\mathcal{X}_{i}\\})(k,m)&=\sum_{n=0}^{N-1}\mathcal{X}(n)g(n-m)e^{\frac{-j2\pi
kn}{N}}\end{split}$ (2)

where $k$ and $m$ are the time index and frequency index, respectively. The
time-frequency responses $\\{s_{i}\\}_{i=1}^{i=n}$ is passed through
transformations $\mathbb{T}^{2}$, (i.e. $\mathbb{T}^{2}_{1}$ and
$\mathbb{T}^{2}_{2}$ are global and local transformations). The SSL KD in this
task is performed by Time Cutout (TC) and Frequency Cutout (FC) transformation
$\mathbb{T}^{2}$ as follows:

$\mathbb{T}^{2}:\\{F_{x,y}\\}_{x={t}_{1},y={f}_{1}}^{{t}_{2},{f}_{2}}=0$ (3)

where $\\{F_{x,y}\\}_{x={t}_{1}}^{{t}_{2}}=0$ and
$\\{F_{x,y}\\}_{y={f}_{1}}^{{f}_{2}}=0$ are TC and FC transformations,
respectively.

We utilize SE-ResNet34, a modified version of ResNet [27] as our backbone
network for both $\mathcal{T}^{2}_{\hat{\phi}}$ and
$\mathcal{S}^{2}_{\hat{\theta}}$ which are trained on different parameters of
$\hat{\phi}$ and $\hat{\theta}$.

###  II-C SSL: Down-stream Task

Different from pre-stream task trained on unlabeled data, the down-stream task
is trained on labeled data. Given a signal $\mathcal{X}$, we transfer student
models (i.e., $\mathcal{S}^{1}_{\theta}$, $\mathcal{S}^{2}_{\hat{\theta}}$),
which were unsupervised trained by the previous module, to extract ECG
characteristic from time-series
($f_{1}=\mathcal{S}^{1}_{\theta}(\mathcal{X})$) and time-frequency signal
($f_{2}=\mathcal{S}^{2}_{\hat{\theta}}(\mathcal{X})$). Because features
$f_{1}$ and $f_{2}$ carry different characteristic of $\mathcal{X}$, we
propose a gate fusion mechanism as follows: (i) We first process features
$f_{1}$ and $f_{2}$ through a soft attention [28] to learn impact factors
($w_{1}$, $w_{2}$) of each feature. The output are weighted and denoted as
$\hat{f}_{1}$ and $\hat{f}_{2}$. (ii) We then stack features $\hat{f}_{1}$ and
$\hat{f}_{2}$ and form $f=[\hat{f}_{1}$, $\hat{f}_{2}]$. A multi-layer
perceptron (MLP) and sigmoid are then applied into $f$. (iii) We finally train
the down-stream task with a binary cross entropy loss between the predicted
label output from the sigmoid function and groundtruth label.

##  III Experimental Results

A. Dataset, Metrics We conduct our experiments on PhysioNet 2020 dataset [29].
PhysioNet comprises six ECG datasets and the total number of records are
43,101 for 111 diagnoses. To fairly compare with other existing methods, we
adopt data stratification from [23] and only 25 classes are selected according
to the challenge. Our proposed network is trained using ten-fold cross
validation. For a broader analysis of performance the results are described
under all recommended metrics i.e., AUROC, AUPRC, Accuracy (Acc), F1, F2,
G-score and the challenge metric (ChM) as in [29].

TABLE I: Performance comparison between proposed method and other approaches
on the PhysioNet2020. SSL-T, SSL-S and SSL-TSG are different setups of our
network.

| Model |  AUROC $\uparrow$ | AUPRC$\uparrow$ |  Acc $\uparrow$ |  F1 $\uparrow$ |  F2 $\uparrow$ |  G2 $\uparrow$ | ChM$\uparrow$
---|---|---|---|---|---|---|---|---
Existing Works | [30] | 84.6 | 32.2 | 20.9 | 25.5 | – | – | 21.9
[31] | – | – | – | – | – | – | 23.6
[32] | 80.6 | 26.1 | 11.3 | 26.0 | 30.9 | 12.6 | 24.8
[33] | 82.5 | 32.6 | 33.1 | 28.6 | – | – | 30.5
[34] | 87.2 | – | – | 39.9 | 43.6 | 23.7 | 40.9
[35] | 55.1 | 52.3 | – | 53.8 | – | – | 53.9
[36] | – | – | – | – | – | – | 53.3
[37] | – | – | – | – | – | – | 58.5
[38] | – | – | – | 60.3 | – | – | 62.7
Our | SSL-T | 95.7 | 64.8 | 48.8 | 61.4 | 64.9 | 40.2 | 64.7
SSL-S | 93.8 | 56.3 | 40.7 | 53.2 | 57.4 | 33.9 | 58.0
SSL-TSG | 96.0 | 65.8 | 48.9 | 62.1 | 65.9 | 41.0 | 65.4

B. Performance and Comparison We conduct a comparison between our proposed
method with other existing approaches as shown in Table I. In this experiment,
we report the performance of our proposed SSL with three different setups: (i)
SSL-T: Work on one modality of time-series data. The network is
$\mathcal{S}^{1}_{\theta}$. There is no gate fusion used; (ii) SSL-S: Work on
one modality of time-frequency data. The network is
$\mathcal{S}^{2}_{\hat{\theta}}$. There is no gate fusion used; (iii) SSL-TSG:
Our proposed method. Both modalities are used. Gate fusion is needed to fuse
$f_{1}$ and $f_{2}$. Our proposed method obtains the best performance and it
gains significant margins on various metrics compared with the existing
methods.

The ablation study in Table II contains two groups corresponding to the
utilization of SSL or not. In the case of w/o SSL, the networks
$\mathcal{S}^{1}_{\theta}$ and $\mathcal{S}^{2}_{\hat{\theta}}$ are not pre-
trained by self-KD. At each group, we test the network on various settings:
(i) T: One modality of time-series data with network
$\mathcal{S}^{1}_{\theta}$. (ii) S: One modality of time-frequency data with
network $\mathcal{S}^{2}_{\hat{\theta}}$. (iii) TSC: Both modalities with both
networks $\mathcal{S}^{1}_{\theta}$ and $\mathcal{S}^{2}_{\hat{\theta}}$ and
there is no gate fusion used. The feature $f_{1}$ and $f_{2}$ are combined by
concatenation; (iv) TSG: Both modalities with both networks
$\mathcal{S}^{1}_{\theta}$ and $\mathcal{S}^{2}_{\hat{\theta}}$. The feature
$f_{1}$ and $f_{2}$ are combined by our proposed gate fusion mechanism. On
each group of with or without SSL, even time-series outperforms time-
frequency, the combination of both modalities with our gate fusion yields the
best performance. The effectiveness of SSL (i.e self-KD) can be clearly seen
when comparing the results with or without SSL on the same modality.

TABLE II: Ablation study of proposed method with different settings. In the
$2^{nd}$ group, SSL denotes when Self-KD is used.

| AUROC | AUPRC | Acc | F1 | F2 | G2 | ChM
---|---|---|---|---|---|---|---
T | 95.34 | 62.93 | 46.14 | 59.56 | 63.37 | 38.43 | 63.25
S | 93.47 | 54.98 | 39.53 | 52.17 | 56.31 | 32.85 | 56.78
TSC | 95.41 | 63.77 | 47.13 | 60.54 | 64.12 | 39.49 | 63.72
TSG | 95.57 | 64.22 | 48.04 | 61.01 | 64.18 | 39.74 | 64.16
SSL-T | 95.70 | 64.81 | 48.78 | 61.40 | 64.89 | 40.17 | 64.71
SSL-S | 93.76 | 56.29 | 40.68 | 53.22 | 57.44 | 33.93 | 58.01
SSL-TSC | 95.65 | 64.64 | 48.13 | 60.92 | 64.33 | 39.73 | 64.37
SSL-TSG | 95.99 | 65.84 | 48.91 | 62.07 | 65.95 | 40.98 | 65.39

Transformations are crucial to the success of the adopted SSL [14]; hence, we
further conduct an ablation study on different transformations as shown in
Fig. 3. On each domain, a baseline is defined when no transformation is used.
In time series domain, we report the performance on transformation of TC, GN
and $\mathbb{T}^{1}$ as defined in Subsec.II-B.1. In time-frequency domain, we
report the performance on transformation of TC, FC and $\mathbb{T}^{2}$ as
defined in Subsec.II-B.2. The baseline obtains good performance – thanks to
the inherent temporal dependency modeling in our network design. Given a long
ECG record, most of existing work pre-processed the ECG signal by partitioning
it into a set of overlapping segments. In such approaches, the temporal
relationship between segments mainly depends on the overlap threshold, which
is typically set as a pre-defined parameter. Unlike the existing work, we
randomly crop the ECG record into a set of segments at each iteration, thus,
the overlap between segments is more flexible and the inter-segment coherence
is modeled better.

Figure 3: Ablation study on different transformation setup on each modality.

## Conclusion

In this paper, we have proposed an SSL-based multimodality network for multi-
lead ECG classification. Our approach includes two modules, one for the pre-
stream task to pre-train networks on unlabeled data, and the other on the
down-stream task to fine-tune the networks on labeled data. Our SSL is built
based on a self-KD mechanism. We have investigated two modalities
corresponding respectively to time-series data, and spectrogram in time-
frequency domain. To combine features from multiple modalities, we have
introduced a gate fusion mechanism. The experimental results and an extensive
ablation study have shown that SSL on multi-modality data is an effective
approach to classify multi-lead ECG.

Acknowledgments: This material is based upon work supported in part by the US
National Science Foundation, under Award No. OIA-1946391, NSF 1920920.

## References

  * [1] V Chouhan and S Mehta.  Threshold-based detection of p and t-wave in ecg using new feature signal.  Com.Sci.& Net.Sec., 8(2):144–153, 2008.
  * [2] M Zabihi, A Rad, et al.  Detection of atrial fibrillation in ecg hand-held devices using a random forest classifier.  In CinC, pages 1–4, 2017.
  * [3] Kashu Yamazaki, Viet-Khoa Vo-Ho, Darshan Bulsara, and Ngan Le.  Spiking neural networks and their applications: A review.  Brain Sciences, 12(7):863, 2022.
  * [4] Ngan Le, Vidhiwar Singh Rathour, Kashu Yamazaki, Khoa Luu, and Marios Savvides.  Deep reinforcement learning in computer vision: a comprehensive survey.  Artificial Intelligence Review, pages 1–87, 2021.
  * [5] S Kevin Zhou, Hoang Ngan Le, Khoa Luu, Hien V Nguyen, and Nicholas Ayache.  Deep reinforcement learning in medical imaging: A literature review.  Medical image analysis, 73:102193, 2021.
  * [6] P Rajpurkar, A.Y Hannun, et al.  Cardiologist-level arrhythmia detection with convolutional neural networks.  arXiv preprint arXiv:1707.01836, 2017.
  * [7] M.D Le, V.S Rathour, et al.  Multi-module recurrent convolutional neural net with transformer encoder for ecg arrhythmia classification.  In BHI, pages 1–5. IEEE, 2021.
  * [8] A.H Ribeiro, M.Horta Ribeiro, G.M.M Paixão, D.M Oliveira, P.R Gomes, J.A Canazart, M.P.S Ferreira, C.R Andersson, P.W Macfarlane, W Meira Jr, et al.  Automatic diagnosis of the 12-lead ecg using a deep neural network.  Nature communications, 11(1):1–9, 2020.
  * [9] U.B Baloglu, M Talo, O Yildirim, R San Tan, and U.R Acharya.  Classification of myocardial infarction with multi-lead ecg signals and deep cnn.  Pattern Recognition Letters, 122:23–30, 2019.
  * [10] R Zhang, P Isola, and A.A Efros.  Colorful image colorization.  In ECCV, pages 649–666. Springer, 2016.
  * [11] G Wang, C Zhang, Y Liu, H Yang, D Fu, H Wang, and P Zhang.  A global and updatable ecg beat classification system based on recurrent neural networks and active learning.  Information Sciences, 501:523–542, 2019.
  * [12] J Huang, B Chen, B Yao, and W He.  Ecg arrhythmia classification using stft-based spectrogram and convolutional neural network.  IEEE access, 7:92871–92880, 2019.
  * [13] Ö Yildirim.  A novel wavelet sequence based on deep bidirectional lstm net model for ecg signal classification.  CBM, 96:189–202, 2018.
  * [14] M Caron, H Touvron, I Misra, H Jégou, J Mairal, P Bojanowski, and A Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF ICCV, pages 9650–9660, 2021.
  * [15] M Caron, P Bojanowski, A Joulin, and M Douze.  Deep clustering for unsupervised learning of visual features.  In ECCV, pages 132–149, 2018.
  * [16] N Srivastava, E Mansimov, and R Salakhudinov.  Unsupervised learning of video representations using lstms.  In ICML, pages 843–852. PMLR, 2015.
  * [17] D Pathak, R Girshick, P Dollár, T Darrell, and B Hariharan.  Learning features by watching objects move.  In CVPR, pages 2701–2710, 2017.
  * [18] M Tran, L Ly, B.S Hua, and N Le.  Ss-3dcapsnet: Self-supervised 3d capsule networks for medical segmentation on less labeled data.  IEEE ISBI, 2022.
  * [19] N Sayed, B Brattoli, and B Ommer.  Cross and learn: Cross-modal self-supervision.  In GCPR, pages 228–243, 2018.
  * [20] D.Q Vu, N.T.H Le, and J.C Wang.  Self-supervised learning via multi-transformation classification for action recognition.  arXiv preprint arXiv:2102.10378, 2021.
  * [21] D Kiyasseh, T Zhu, and D.A Clifton.  Clocs: Contrastive learning of cardiac signals.  arXiv preprint arXiv:2005.13249, 2020.
  * [22] X Lan, D Ng, S Hong, and M Feng.  Intra-inter subject self-supervised learning for multivariate cardiac signals.  arXiv preprint arXiv:2109.08908, 2021.
  * [23] T Mehari and N Strodthoff.  Self-supervised representation learning from 12-lead ecg data.  CBM, 141:105114, 2022.
  * [24] Duc-Quang Vu, Ngan Le, and Jia-Ching Wang.  Teaching yourself: A self-knowledge distillation approach to action recognition.  IEEE Access, 9:105711–105723, 2021.
  * [25] G Hinton, O Vinyals, J Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2(7), 2015.
  * [26] T He, Z Zhang, H Zhang, X Zhang, J Xie, and M Li.  Bag of tricks for image classification with convolutional neural networks.  In CVPR, pages 558–567, 2019.
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [28] K Xu, J Ba, R Kiros, K Cho, A Courville, R Salakhudinov, R Zemel, and Y Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In ICML, pages 2048–2057. PMLR, 2015.
  * [29] E.A.P Alday, A Gu, A.J Shah, C Robichaux, A.K.I Wong, C Liu, F Liu, A.B Rad, A Elola, S Seyedi, et al.  Classification of 12-lead ecgs: the physionet/computing in cardiology challenge 2020.  Physiological measurement, 41(12):124003, 2020.
  * [30] P.S Ignacio, J.A Bulauan, and J.R Manzanares.  A topology informed random forest classifier for ecg classification.  In 2020 CinC, pages 1–4, 2020.
  * [31] R Duan, X He, and Z Ouyang.  Madnn: A multi-scale attention deep neural net for arrhythmia classification.  In 2020 CinC, pages 1–4, 2020.
  * [32] A.W Wong, A Salimi, et al.  Multilabel 12-lead electrocardiogram classification using beat to sequence autoencoders.  In ICASSP, pages 1270–1274. IEEE, 2021.
  * [33] D Nankani, P Saikia, and R.D Baruah.  Automatic concurrent arrhythmia classification using deep residual neural networks.  In 2020 CinC, pages 1–4, 2020.
  * [34] B.J Singstad and C Tronstad.  Convolutional neural net and rule-based algorithms for classifying 12-lead ecgs.  In 2020 CinC, pages 1–4, 2020.
  * [35] Y Feng and E Vigmond.  Deep multi-label multi-instance classification on 12-lead ecg.  In 2020 CinC, pages 1–4, 2020.
  * [36] A Natarajan, Y Chang, et al.  A wide and deep transformer neural network for 12-lead ecg classification.  In 2020 CinC, pages 1–4, 2020.
  * [37] N Nonaka and J Seita.  Electrocardiogram classification by modified efficientnet with data augmentation.  In 2020 CinC, pages 1–4, 2020.
  * [38] Z Jiang, T.P Almeida, et al.  Diagnostic of multiple cardiac disorders from 12-lead ecgs using graph convolutional net based multi-label classification.  In 2020 CinC, pages 1–4, 2020.