# Self-Ensembling with GAN-based Data Augmentation for Domain Adaptation in
Semantic Segmentation

Jaehoon Choi
KAIST
Taekyung Kim
KAIST
Changick Kim
KAIST
{whdns44, tkkim93<EMAIL_ADDRESS>
###### Abstract

Deep learning-based semantic segmentation methods have an intrinsic limitation
that training a model requires a large amount of data with pixel-level
annotations. To address this challenging issue, many researchers give
attention to unsupervised domain adaptation for semantic segmentation.
Unsupervised domain adaptation seeks to adapt the model trained on the source
domain to the target domain. In this paper, we introduce a self-ensembling
technique, one of the successful methods for domain adaptation in
classification. However, applying self-ensembling to semantic segmentation is
very difficult because heavily-tuned manual data augmentation used in self-
ensembling is not useful to reduce the large domain gap in the semantic
segmentation. To overcome this limitation, we propose a novel framework
consisting of two components, which are complementary to each other. First, we
present a data augmentation method based on Generative Adversarial Networks
(GANs), which is computationally efficient and effective to facilitate domain
alignment. Given those augmented images, we apply self-ensembling to enhance
the performance of the segmentation network on the target domain. The proposed
method outperforms state-of-the-art semantic segmentation methods on
unsupervised domain adaptation benchmarks.

##  1 Introduction

Semantic segmentation has been widely studied in the computer vision field.
Its goal is to assign image category labels to each pixel in the image. A wide
variety of algorithms based on deep neural networks have achieved high
performance with sufficient amounts of annotated datasets. However, creating
large labeled datasets for semantic segmentation is cost-expensive and time-
consuming [7]. To overcome the annotation burden, researchers utilize modern
computer graphics to easily generate synthetic images with ground truth labels
[36]. Unfortunately, in practice, models trained with synthetic data do not
perform well on a realistic domain because there exists a distribution
difference called domain shift. Unsupervised domain adaptation handles the
domain shift by transferring knowledge from the labeled dataset in the source
domain to the unlabeled dataset in the target domain [3].

Figure 1: The overall framework of our method. Given labeled synthetic data
and unlabeled real data, we propose a Target-Guided and Cycle-Free Data
Augmentation (TGCF-DA) method to generate labeled augmented data (green). We
introduce two segmentation networks as the teacher and the student in order to
implement the self-ensembling algorithm (orange). Both segmentation networks
are trained by augmented data as well as synthetic and real data. During the
learning process, the teacher network transfers its knowledge to the student
network.

Recent approaches for domain adaptation focus on aligning features extracted
from the source and target data. In particular, most of the domain adaptation
methods in semantic segmentation depend on adversarial training aiming to
minimize the domain discrepancy through domain confusion [15, 14, 44, 41, 16,
52]. However, adversarial approaches suffer from a significant drawback. Since
these methods seek to align the global distributions of two different domains,
the adversarial loss may trigger a negative transfer, which aligns the target
feature with the source feature in an incorrect semantic category. The
negative transfer can have adverse effect on features that are already well
aligned. Thus, this adaptation often performs even worse than a network
trained solely on the source domain. Instead of adversarial training, we take
an alternative way to perform feature-level domain alignment. We adopt self-
ensembling [9], one of the effective methods for domain adaptation in
classification.

Self-ensembling is composed of a teacher and a student network, where the
student is compelled to produce consistent predictions provided by the teacher
on target data. As the teacher is an ensembled model that averages the
student’s weights, predictions from the teacher on target data can be thought
of as the pseudo labels for the student. While recent self-ensembling proves
its effectiveness in classification, these approaches require heavily-tuned
manual data augmentation [9] for successful domain alignment. Furthermore,
although such data augmentation consisting of various geometric
transformations is effective in classification, it is not suited to minimize
the domain shift in semantic segmentation. Two different geometric
transformations on each input can cause spatial misalignment between the
student and teacher predictions. Thus, we propose a novel data augmentation
method to deal with this issue.

Our augmented image synthesis method is based on generative adversarial
networks (GANs) [12]. We aim to generate augmented images, in which semantic
contents are preserved, because these images with the inconsistent semantic
content impair the segmentation performance due to the pixel-level
misalignment between augmented images and source labels. Hence, we add a
semantic constraint for the generator to preserve global and local structures,
_i.e_. the semantic consistency. Furthermore, we propose a target-guided
generator, which produces images conditioned on style information extracted
from the target domain. In other words, our generator synthesizes augmented
images maintaining semantic information, while only transferring styles from
target images.

Most previous studies for GAN-based Image-to-Image translation methods [53,
49, 27, 25, 21, 19, 31] rely on various forms of cycle-consistency. However,
incorporating cycle-consistency into unsupervised domain adaptation has two
limitations. First, it needs redundant modules such as a target-to-source
generator and corresponding computational burden. Second, cycle-consistency
may be too strong when target data are scarce compared to source data [17],
which is the general setting of unsupervised domain adaptation. Our proposed
model does not consider all kinds of cycle-consistency. We refer to our method
as Target-Guided and Cycle-Free Data Augmentation (TGCF-DA).

Our universal framework is illustrated in Fig. 1. We employ TGCF-DA to produce
augmented images. Then, the segmentation network learns from the source,
target and augmented data through self-ensembling. The main contributions of
this paper are summarized as follows:

  * •

We propose a novel data augmentation method with a target-guided generator and
a cycle-free loss which is more efficient and suitable for semantic
segmentation in unsupervised domain adaptation.

  * •

We build a unified framework that collaborates the self-ensembling with TGCF-
DA.

  * •

Our approach achieves the state-of-the-art performances on challenging
benchmark datasets. Also, we conduct extensive experiments and provide
comprehensive analyses for the proposed method.

##  2 Related work

Unsupervised Domain Adaptation for Semantic Segmentation: Recently
unsupervised domain adaptation for semantic segmentation has received much
attention. The first attempt to this task is FCNs in the wild [15], which
simultaneously performs the global and local alignment with adversarial
training. Adversarial training is the predominant approach focusing on a
feature-level adaptation to generate domain-invariant features through domain
confusion, e.g., [6, 5, 44, 41, 16, 39, 18]. This idea is extended to jointly
adapt representations at both pixel and feature level through various
techniques such as cycle-consistency loss [14, 34] or style transfer [8, 47].
Except for adversarial training methods, there is a different approach based
on self-training. CBST [57] introduces self-training to produce pseudo labels
and retrain the network with these labels.

Self-Ensembling: Self-Ensembling [56, 38] is proposed in the field of semi-
supervised learning. A popular method for semi-supervised learning is the
consistency regularization, which employs unlabeled data to produce consistent
predictions under perturbations [40, 2]. Laine and Aila [24] propose Temporal
Ensembling using a per-sample moving average of predictions for the consistent
output. Tarvainen and Valpola [43] suggest an exponential moving average of
the model weights instead of average of predictions. The self-ensembling
method [9] applies a Mean Teacher framework to unsupervised domain adaptation
with some modifications. In [35], Perone _et al_. address medical imaging
segmentation tasks by applying the self-ensembling method akin to the previous
method. Yonghao _et al_. [48] utilize the self-ensembling attention network to
extract attention-aware features for domain adaptation.

Figure 2: An overview of the proposed framework. 1) The source and target
images are fed into Target-Guided generator to produce augmented images. 2)
The supervised loss is a multi-class cross-entropy loss with source images and
augmented images. 3) The consistency loss is a mean squared error between both
prediction maps extracted from the student and teacher network. 4) A total
training loss is the weighted sum of the supervised loss and the consistency
loss. 5) We perform data augmentation only for target samples to complement
the consistency loss. 6) The teacher network’s weights are the exponential
moving average (EMA) of those of the student network.

Image-to-Image Translation: Recent approaches for Image-to-Image (I2I)
Translation are based on Generative Adversarial Networks (GANs) [12]. In the
case of unpaired training images, one popular constraint is cycle-consistency
that maps a given image to the target domain and reconstructs the original
image [23, 53, 49]. UNIT [27] introduces a constraint for learning a shared
latent space. However, all the aforementioned methods suffer from a lack of
diversity in translated images. To produce multi-modal outputs, one possible
approach injects noise vectors as additional inputs to the generator [54, 1,
11], but it could lead to the mode collapse problem. Also, Since cycle-
consistency is too restrictive, variants of cycle-consistency [54, 21, 25] are
developed for multi-modal I2I translation. A different approach is to apply
neural style transfer [10, 45, 22, 20]. In particular, concurrent works [21,
31] employ an adaptive instance normalization [20] to transfer style from the
exemplar to the original image. In addition, the authors of AugGAN [19]
exploit the segmentation information for improving I2I translation network.
Our task is entirely different from AugGAN because domain adaptation cannot
use segmentation labels of the target data.

##  3 Proposed Method

In this work, we introduce the unified framework, which is built upon the
self-ensembling for semantic segmentation. The key to improve the capacity of
the self-ensembling for semantic segmentation is the GAN-based data
augmentation to align representations of source and target rather than
geometric transformations mostly used in existing self-ensembling for
classification. To achieve this goal, we present a novel Target-Guided and
Cycle-Free Data Augmentation (TGCF-DA) with a target-guided generator and a
semantic constraint. The target-guided generator translates source images to
different styles in the target domain. Our student network learns the source
images and augmented images from TGCF-DA with a supervised loss by computing
cross-entropy loss. Also, we only use target samples to compute the
consistency loss, which is defined as the mean squared error between
prediction maps generated from the student and teacher networks.

More formally, let $X_{\mathcal{S}}$ and $X_{\mathcal{T}}$ denote the source
domain and target domain. We have access to $N_{s}$ labeled source samples
$\\{(x_{s}^{i},y_{s}^{i})\\}_{i=1}^{N_{s}}$ with $x_{s}^{i}\in
X_{\mathcal{S}}$ and the corresponding label maps $y_{s}^{i}$. The target
domain has $N_{t}$ unlabeled target samples $\\{x_{t}^{i}\\}_{i=1}^{N_{t}}$,
where $x_{t}^{i}\in X_{\mathcal{T}}$. $P_{\mathcal{S}}$ and $P_{\mathcal{T}}$
denote the source and target data distributions, respectively. The source and
target data share $C$ categories. Let $f_{S}$ and $f_{T}$ be a student
segmentation network and a teacher segmentation network.

###  3.1 Target-guided generator

Based on the assumption that image can be decomposed into two disentangled
representations [27, 21], a content and a style, we adopt a source encoder for
generating content representation and a target encoder for extracting style
representation. To combine these two representations properly, we apply
Adaptive Instance Normalization (AdaIN) [20] to feature maps of source images.
As in [21], the target encoder with multiple fully connected layers provide
the learnable affine transformation parameters $(\gamma_{t},\beta_{t})$ to
normalize the feature maps of a source image for each channel. The AdaIN
operation is defined as:

$\tilde{F}_{s}^{i}=\gamma_{t}^{i}\>(\frac{F_{s}^{i}-\mu(F_{s}^{i})}{\sigma(F_{s}^{i})})+\beta_{t}^{i}\>,$
(1)

where $F_{s}^{i}$ denotes the source feature map for the $i$-th channel.
$\mu(\cdot)$ and $\sigma(\cdot)$ respectively denote mean and variance across
spatial dimensions. Our generator is guided by the style information of target
samples through AdaINs at intermediate residual blocks while preserving the
spatial structure of source images, _i.e_. the semantic consistency of source
images is retained.

Figure 3: The overview of TGCF-DA based on GAN [12]. The blue box describes
the target-guided generator $G$. The red box is the pretrained segmentaion
model $f_{seg}$ with fixed weights. The purple box is the discriminator $D$.

###  3.2 Semantic constraint

We utilize a semantic constraint to preserve semantic content at pixel level.
Given the labeled source data, we can pretrain the segmentation model such as
FCN-8s [29] for constraining the generator. The pretrained segmentation model
$f_{seg}$ with fixed weights encourages the semantic consistency between the
images before and after the translation. Thanks to this semantic constraint,
our network can preserve the objects in images without distortion.
Furthermore, this constraint is crucial to stabilizing the adversarial
training without the cycle-consistency. Since the cycle-consistency enforces
the strict constraint for matching two distributions, it is effective to
prevent the mode collapse and to stabilize the adversarial training [26].
Without the cycle-consistency, our adversarial training is vulnerable to
instability of GAN training. However, this semantic constraint guarantees
stable adversarial training by strongly enforcing semantic consistency. We
define the semantic constraint loss as the cross-entropy loss:

$\begin{split}L_{sem}&(f_{seg},G)=\\\ &\frac{-1}{HW}\sum_{k=1}^{H\times
W}\sum_{c=1}^{C}y_{s}^{(k,c)}\>log(f_{seg}(G(x_{s},x_{t}))^{(k,c)})\>,\\\
\end{split}$ (2)

where $G(x_{s},x_{t})$ is the generated image of size $H$$\times$$W$ produced
by the target-guided generator G.

###  3.3 Target-guided and cycle-free data augmentation

We introduce a GAN designed for Target-Guided and Cycle-Free Data Augmentation
(TGCF-DA). As in Fig. 3, $G$ is the target-guided generator and $D$ is the
discriminator proposed in [46]. We use the adversarial objective from LSGAN
[32] and apply the spectral normalization [33] to stabilize the GAN training.
The GAN loss is defined as:

$\begin{split}L_{GAN}(G,D)&=E_{(x_{s},x_{t})\sim(P_{\mathcal{S}},P_{\mathcal{T}})}[D(G(x_{s},x_{t}))^{2}]\\\
&+E_{x_{t}\sim P_{\mathcal{T}}}[(D(x_{t})-1)^{2}]\>.\\\ \end{split}$ (3)

This loss ensures that $G$ produces new images visually similar to target
images without losing semantic content in source images. Since the
segmentation model $f_{seg}$ is fixed, we jointly train the target-guided
generator and discriminator to optimize the overall loss:

$\begin{split}L_{TGCF-DA}=L_{GAN}+\>\lambda_{sem}L_{sem}\>,\end{split}$ (4)

where $\lambda_{sem}$ is a weight to balance the contribution of the GAN loss
and semantic constraint. The pretrained target-guided generator is employed to
synthesize augmented images with the purpose of data augmentation in the self-
ensembling.

###  3.4 Self-ensembling

We construct the teacher network $f_{T}$ and the student network $f_{S}$. The
teacher’s weights $t_{i}$ at training step $i$ are updated by the student’s
weights $s_{i}$ following the formula:

$t_{i}=\alpha t_{i-1}+(1-\alpha)s_{i}\>,$ (5)

where $\alpha$ is an exponential moving average decay. During training, each
mini-batch consists of source samples, augmented samples, and target samples.
We use source samples and augmented samples to compute the supervised loss
$L_{sup}$, which is cross-entropy function for semantic segmentation. This
loss function enables the student network to produce the semantically accurate
prediction for the source and augmented samples. The consistency loss
$L_{con}$ is formulated as the mean-squared error between the prediction maps
generated from the student and teacher network:

$\begin{split}L_{con}(f_{S},f_{T})=E_{x_{t}\sim
P_{\mathcal{T}}}[\|\sigma(f_{S}(x_{t}))-\sigma(f_{T}(x_{t}))\|^{2}]\>,\end{split}$
(6)

where $\sigma$ is a softmax function to compute probability of prediction
maps. The total loss $L_{total}$ is the weighted sum of the supervised loss
$L_{sup}$ and the consistency loss $L_{con}$:

$\begin{split}L_{total}=L_{sup}+\delta_{con}L_{con}\>,\end{split}$ (7)

where $\delta_{con}$ is the weight of consistency loss subject to the ramp-
ups. Contrary to [9], we empirically observe that weight ramp-up is necessary
for enhancing the effectiveness of the consistency loss.

###  3.5 Data augmentation for target samples

Here, data augmentation for target samples is not relevant to TGCF-DA. This
data augmentation is only applied to target samples in order to compute the
consistency loss for the self-ensembling in Section 3.4. In classification
[9], the goal of random data augmentations for target samples is forcing the
student network to produce different predictions for the same target sample.
Aforementioned above, image-level transformations such as geometric
transformations are not helpful for the pixel-level prediction task like
semantic segmentation [28]. Thus, we inject Gaussian noise to target samples,
which are fed to student and target networks respectively. In addition, we
apply Dropout [42] for weight perturbation. As a result, our student network
is forced to produce consistent predictions with the teacher network under
different perturbations for target samples and parameters of each network.

##  4 Experiments

This section describes experimental setups and details of synthetic-to-real
domain adaptation. Then, we will report the experiment results compared with
the previous researches. Furthermore, we will provide ablation studies to
validate the effectiveness of our method.

###  4.1 Datasets

For a synthetic source domain, we used SYNTHIA [37] and GTA5 [36] datasets.
Then, we evaluated our method on Cityscapes dataset [7] as a real-world target
domain following similar settings in [15, 51, 44, 41]. We briefly introduce
the details of datasets as following:

GTA5. GTA5 [36] contains 24966 urban scene images with pixel-level
annotations. These high-resolution images are rendered from the gaming engine
Grand Theft Auto V. Following [15], we used the 19 categories of the
annotations compatible with those of the Cityscapes. We randomly picked 1000
images from GTA5 for validation purpose.

SYNTHIA. SYNTHIA [37] is a large-scale dataset of video sequences rendered
from a virtual city. We used SYNTHIA-RAND-CITYSCAPES, consisting of 9400
images with pixel-level annotations. Inheriting from the previous work [51],
we chose 16 categories common in both SYNTHIA and Cityscapes. We randomly
selected 100 images for evaluation.

Cityscapes. Cityscapes [7] contains urban street scenes collected from 50
cities around Germany and neighboring countries. It has a training set with
2975 images and a validation set with 500 images.

We can utilize source images and labels from either SYNTHIA or GTA5, as well
as target images without labels from the training set of Cityscapes. The
validation set in Cityscapes is treated as the evaluation set for our domain
adaptation experiment. We report IoU (Intersection-over-Union) for each class
and mIoU (mean IoU) to measure the segmentation performance. In supplementary
material, we provide additional experimental results on the BDD100K dataset
[50].

###  4.2 Experiment setup and implementation details

TGCF-DA. Our augmentation network for TGCF-DA is composed of the generator,
the discriminator and the segmentation model. The generator is built upon the
auto-encoder architecture used by MUNIT [21], but modified to act as the
cycle-free generator. It consists of the source encoder, the target encoder
and the decoder. The source encoder includes strided convolutional layers to
downsample the source images and residual blocks [13] to compute the content
representations. The decoder consists of residual blocks and transposed
convolutional layers to upsample the combined representations. The target
encoder is comprised of strided convolutional layers and fully connected
layers to provide the style representations. Multi-scale discriminators
described in [46] are employed as our discriminator. We set the weight
$\lambda_{sem}$ to 10 in all experiments.

Self-Ensembling. In all our experiments, we employed a VGG-16 backbone for our
semantic segmentation network. Following Deeplab [4], we incorporated ASPP
(Atrous Spatial Pyramid Pooling) as the decoder and then used an upsampling
layer to get the final segmentation output. Before the upsampling layer, the
output of the final classifier is used to compute the consistency loss in
Section 3.4. Motivated by [43], we utilized the sigmoid ramp-up for the
consistency loss weight $\delta_{con}$. The details of the consistency loss
weight is analyzed in Section 5.3. During training process, the images are
resized and cropped to 480$\times$960 resolution, and for evaluation we
upsample our prediction maps to 1024$\times$2048 resolution. The details of
our architecture and experiments will be available in the supplementary
material.

###  4.3 Experimental results

(a) GTA5 $\rightarrow$ Cityscapes
---
Method |  Mech. |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign |  vegetation |  terrain |  sky |  person |  rider |  car |  truck |  bus |  train |  motorbike |  bike |  mIoU
Baseline (Source Only) | - | 61.0 | 18.5 | 66.2 | 18.0 | 19.6 | 19.1 | 22.4 | 15.5 | 79.6 | 28.5 | 58.0 | 44.5 | 1.7 | 66.6 | 14.1 | 1.1 | 0.0 | 3.2 | 0.7 | 28.3
Curriculum DA [51] | ST | 72.9 | 30.0 | 74.9 | 12.1 | 13.2 | 15.3 | 16.8 | 14.1 | 79.3 | 14.5 | 75.5 | 35.7 | 10.0 | 62.1 | 20.6 | 19.0 | 0.0 | 19.3 | 12.0 | 31.4
CyCADA [14] | AT | 85.2 | 37.2 | 76.5 | 21.8 | 15.0 | 23.8 | 22.9 | 21.5 | 80.5 | 31.3 | 60.7 | 50.5 | 9.0 | 76.9 | 17.1 | 28.2 | 4.5 | 9.8 | 0.0 | 35.4
MCD [39] | AT | 86.4 | 8.5 | 76.1 | 18.6 | 9.7 | 14.9 | 7.8 | 0.6 | 82.8 | 32.7 | 71.4 | 25.2 | 1.1 | 76.3 | 16.1 | 17.1 | 1.4 | 0.2 | 0.0 | 28.8
LSD-seg [41] | AT | 88.0 | 30.5 | 78.6 | 25.2 | 23.5 | 16.7 | 23.5 | 11.6 | 78.7 | 27.2 | 71.9 | 51.3 | 19.5 | 80.4 | 19.8 | 18.3 | 0.9 | 20.8 | 18.4 | 37.1
AdaptSegNet [44] | AT | 87.3 | 29.8 | 78.6 | 21.1 | 18.2 | 22.5 | 21.5 | 11.0 | 79.7 | 29.6 | 71.3 | 46.8 | 6.5 | 80.1 | 23.0 | 26.9 | 0.0 | 10.6 | 0.3 | 35.0
ROAD [5] | AT | 85.4 | 31.2 | 78.6 | 27.9 | 22.2 | 21.9 | 23.7 | 11.4 | 80.7 | 29.3 | 68.9 | 48.5 | 14.1 | 78.0 | 19.1 | 23.8 | 9.4 | 8.3 | 0.0 | 35.9
Conservative Loss [55] | AT | 85.6 | 38.3 | 78.6 | 27.2 | 18.4 | 25.3 | 25.0 | 17.1 | 81.5 | 31.3 | 70.6 | 50.5 | 22.3 | 81.3 | 25.5 | 21.0 | 0.1 | 18.9 | 4.3 | 38.1
DCAN [47] | SR | 82.3 | 26.7 | 77.4 | 23.7 | 20.5 | 20.4 | 30.3 | 15.9 | 80.9 | 25.4 | 69.5 | 52.6 | 11.1 | 79.6 | 24.9 | 21.2 | 1.3 | 17.0 | 6.7 | 36.2
CBST [57] | ST | 66.7 | 26.8 | 73.7 | 14.8 | 9.5 | 28.3 | 25.9 | 10.1 | 75.5 | 15.7 | 51.6 | 47.2 | 6.2 | 71.9 | 3.7 | 2.2 | 5.4 | 18.9 | 32.4 | 30.9
Self-Ensembling (SE) | ST | 76.4 | 16.7 | 71.5 | 13.0 | 13.1 | 17.5 | 17.3 | 8.3 | 76.5 | 16.3 | 67.4 | 42.5 | 10.4 | 78.1 | 27.9 | 37.2 | 0.0 | 22.2 | 7.4 | 32.6
TGCF-DA | AT | 73.9 | 19.8 | 74.8 | 19.7 | 21.8 | 20.7 | 26.7 | 12.4 | 78.0 | 22.3 | 72.0 | 53.4 | 12.9 | 73.3 | 24.5 | 28.5 | 0.0 | 24.3 | 14.1 | 35.4
Ours (TGCF-DA + SE) | AT+ST | 90.2 | 51.5 | 81.1 | 15.0 | 10.7 | 37.5 | 35.2 | 28.9 | 84.1 | 32.7 | 75.9 | 62.7 | 19.9 | 82.6 | 22.9 | 28.3 | 0.0 | 23.0 | 25.4 | 42.5
Target Only | - | 94.3 | 77.7 | 86.6 | 52.9 | 50.4 | 50.1 | 52.9 | 57.0 | 81.4 | 64.8 | 94.1 | 57.8 | 55.5 | 87.6 | 79.0 | 56.1 | 19.6 | 45.3 | 20.9 | 62.3

(b) SYNTHIA $\rightarrow$ Cityscapes
---
Method |  Mech. |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign |  vegetation |  sky |  person |  rider |  car |  bus |  motorbike |  bike |  mIoU |  mIoU*
Baseline (Source Only) | - | 6.8 | 15.4 | 56.8 | 0.8 | 0.1 | 14.6 | 4.7 | 6.8 | 72.5 | 78.6 | 41.0 | 7.8 | 46.9 | 4.7 | 1.8 | 2.1 | 22.6 | 24.1
Curriculum DA [51] | ST | 65.2 | 26.1 | 74.9 | 0.1 | 0.5 | 10.7 | 3.7 | 3.0 | 76.1 | 70.6 | 47.1 | 8.2 | 43.2 | 20.7 | 0.7 | 13.1 | 29.0 | 34.8
LSD-seg [41] | AT | 80.1 | 29.1 | 77.5 | 2.8 | 0.4 | 26.8 | 11.1 | 18.0 | 78.1 | 76.7 | 48.2 | 15.2 | 70.5 | 17.4 | 8.7 | 16.7 | 36.1 | -
AdaptSegNet [44] | AT | 78.9 | 29.2 | 75.5 | - | - | - | 0.1 | 4.8 | 72.6 | 76.7 | 43.4 | 8.8 | 71.1 | 16.0 | 3.6 | 8.4 | - | 37.6
ROAD [5] | AT | 77.7 | 30.0 | 77.5 | 9.6 | 0.3 | 25.8 | 10.3 | 15.6 | 77.6 | 79.8 | 44.5 | 16.6 | 67.8 | 14.5 | 7.0 | 23.8 | 36.2 | -
Conservative Loss [55] | AT | 80.0 | 31.4 | 72.9 | 0.4 | 0.0 | 22.4 | 8.1 | 16.7 | 74.8 | 72.2 | 50.9 | 12.7 | 53.9 | 15.6 | 1.7 | 33.5 | 34.2 | 40.3
DCAN [47] | SR | 79.9 | 30.4 | 70.8 | 1.6 | 0.6 | 22.3 | 6.7 | 23.0 | 76.9 | 73.9 | 41.9 | 16.7 | 61.7 | 11.5 | 10.3 | 38.6 | 35.4 | -
CBST [57] | ST | 69.6 | 28.7 | 69.5 | 12.1 | 0.1 | 25.4 | 11.9 | 13.6 | 82.0 | 81.9 | 49.1 | 14.5 | 66.0 | 6.6 | 3.7 | 32.4 | 35.4 | 36.1
Self-Ensembling (SE) | ST | 40.1 | 19.6 | 75.2 | 2.6 | 0.2 | 23.2 | 4.0 | 9.8 | 60.3 | 38.3 | 49.1 | 14.0 | 67.0 | 17.4 | 6.4 | 11.9 | 27.5 | 29.2
TGCF-DA | AT | 63.9 | 25.6 | 75.9 | 5.4 | 0.1 | 22.6 | 2.6 | 6.8 | 78.4 | 77.2 | 48.7 | 16.5 | 62.2 | 24.2 | 5.0 | 22.1 | 33.6 | 39.8
Ours (TGCF-DA + SE) | AT+ST | 90.1 | 48.6 | 80.7 | 2.2 | 0.2 | 27.2 | 3.2 | 14.3 | 82.1 | 78.4 | 54.4 | 16.4 | 82.5 | 12.3 | 1.7 | 21.8 | 38.5 | 46.6
Target Only | - | 89.2 | 85.3 | 90.7 | 65.5 | 60.7 | 21.5 | 2.1 | 7.2 | 74.2 | 93.2 | 61.8 | 40.1 | 78.4 | 81.4 | 36.7 | 24.8 | 57.1 | 64.1

Table 1: The semantic segmentation results on Cityscapes validation set when
evaluating the model trained on (a) GTA5 and (b) SYNTHIA. All segmentation
models in table use VGG-16 based models. The mIoU* denotes the segmentation
results over the 13 common classes. “Source Only” denotes the evaluation
result of models only trained on source data. “Target Only” denotes the
segmentation results in supervised settings. The mechanism “AT”, “ST” and “SR”
stand for adversarial training, self-training, and style transfer
respectively.

We report experimental results of the proposed method on two adaptation
experiments in Table 1. We compare our proposed method with Curriculum DA
[51], CyCADA [14], MCD [39], LSD-seg [41], AdaptSegNet [44], ROAD [5],
Conservative Loss [55], DCAN [47], and CBST [57]. In Table 1, Self-Ensembling
(SE) represents the segmentation performance of the network trained by source
and target through the self-ensembling, without our data augmentation method.
TGCF-DA indicates the segmentation network trained by the source data and
augmented data generated from TGCF-DA with corresponding labels. Ours (TGCF-DA
+ SE) denotes our proposed framework comprised of TGCF-DA and the self-
ensembling method. The proposed method significantly outperforms the baseline
by 14.2% on GTA5$\rightarrow$Cityscapes and 13.1% on
SYNTHIA$\rightarrow$Cityscapes. Our method makes further improvement compared
to the source only baseline and also achieves the state-of-the-art mIoU scores
on both experiments.

###  4.4 Ablation studies

Ablation for Self-Ensembling (SE): Comparing the baseline and SE, SE shows
small improvement in mIoUs by 4.3% in Table 1-(a) and by 4.9% in Table 1-(b).
However, in details, we observe that SE does not perform well during the whole
training process as shown in Fig. 4 (blue and orange lines). In contrast to
our proposed method (TCFD-DA + SE), the teacher and student networks do not
maintain complementary correlations.

Ablation for TGCF-DA: TGCF-DA is necessary to generate synthetic data, which
help the network reduce the domain shift. Compared to the baseline, TGCF-DA
improves the mIoUs by 7.1% in Table 1-(a) and by 11.0% in Table 1-(b). Such
improvements validate that TGCF-DA serves as a useful way to reduce the domain
shift. Except for TGCF-DA, SE shows the poor results in both experiments. On
the contrary, our proposed method in Fig. 4 (grey and yellow lines) clearly
demonstrates that the teacher updated by the student continues to improve
segmentation capability, and successfully transfer its knowledge to the
student. As a result, the teacher and student of our method enhance their
performance simultaneously. These results substantiate our intuition that
TGCF-DA enhances the capability of the self-ensembling algorithm for semantic
segmentation.

Figure 4:  The testing mIoUs of SE (blue and orange) and our method (grey and
yellow) $w.r.t$ training epochs on the GTA5 $\rightarrow$ Cityscapes
experiment. Figure 5: Visualization results of GTA5 $\rightarrow$ Cityscapes
(first and second rows) and SYNTHIA $\rightarrow$ Cityscapes (third and fourth
rows). Segmentation results at 10K training steps (first and third rows) and
56K training steps (second and fourth rows). The fourth and fifth columns
illustrate the heatmap of the consistency loss and disagreement map between
the student and teacher networks.

##  5 Analysis

In this section, we provide visualization results and analysis on varaious
components of our proposed framework.

###  5.1 Visualization

The effectiveness of the self-ensembling is visualized in Fig. 5. We validate
that the teacher network generates better predictions, and then different
predictions between the teacher and student networks cause consistency loss to
enforce the consistency of their predictions. In Fig. 5, the first and third
rows show that predictions of the teacher can be a good proxy for training the
student network early in the training. In addition, we point out that the
consistency loss concentrates on the boundary of each object in the later
training stage. Hence, the consistency loss can play a role in refining
boundaries of semantic objects where the segmentation model are likely to
output wrong predictions.

In Fig. 7, we show the example results of TGCF-DA compared with other Image-
to-Image (I2I) translation methods: CycleGAN [53], UNIT [27], and MUNIT [21].
Both CycleGAN and UNIT often generate distorted images containing corrupted
objects and artifacts. MUNIT is capable of preserving objects in images, but
we observe that the style of the majority classes in the target image is often
matched to elements of different classes in the source image, which is similar
to “spills over” problem in [30]. For example, the translated image from MUNIT
shows artifacts in the sky like road texture of the target domain. Compared to
the methods mentioned above, our method is not only computationally cheap and
memory efficient due to the cycle-free loss but also demonstrating compelling
visual results with preserving semantic consistency.

###  5.2 Analysis of self-ensembling with per-class IoUs

Figure 6: Per-class IoU gains through the self-ensembling. The blue bar
represents per-class IoU gains in GTA5$\rightarrow$Cityscapes experiment. The
orange bar indicates the per-class IoU gains in SYNTHIA$\rightarrow$Cityscapes
experiment. Figure 7: Example images of SYNTHIA synthesized in the style of
Cityscapes with CycleGAN [53], UNIT [27], and MUNIT [21].

To better understand the self-ensembling, we compare the per-class IoUs of our
method with and without the self-ensembling. In Fig. 6, we show the per-class
IoU gains between TGCF-DA and Ours (TGCF-DA + SE). Although the IoU scores in
the most categories are generally improved, there is a difference in
performance gains among different categories. Figure 6 demonstrates that the
IoU gains in majority classes (such as “road”) are generally better than those
in minority classes (like “bus”). These experimental results are attributed to
the self-ensembling and class imbalance issues. Due to the class imbalance,
the segmentation network often produces incorrect predictions on minority
classes [57]. In the self-ensembling method, this effect can be strengthened
because the student is iteratively learned from predictions of the teacher,
which tends to make incorrect predictions on minority classes rather than
majority classes. Thus, the self-ensembling gives rise to large improvements
in per class IoUs of majority classes compared to minority classes. It is
worth noting that this result accords with our intuition that predictions of
the teacher network serve as pseudo labels for the student network.

###  5.3 Hyperparameter sensitivity on self-ensembling

|  Ramp-up coefficient $\delta_{0}$ |  EMA decay $\alpha$
---|---|---
1 | 3 | 30 | 50 | 0.9 | 0.99 | 0.999
GTA5 | 41.3 | 42.3 | 42.5 | 33.6 | 37.6 | 38.9 | 42.5
SYN | 35.4 | 36.1 | 38.5 | 32.5 | 36.2 | 38.5 | 37.8
Table 2: Hyperparameter sensitivity. GTA5 denotes GTA5 $\rightarrow$
Cityscapes experiment and SYN denotes SYNTHIA $\rightarrow$ Cityscpaes
experiment.

In the self-ensembling, the consistency loss weight $\delta$ and the
exponential moving average (EMA) decay $\alpha$ are important hyperparameters.
We conduct the experiments to explore the sensitivity of these
hyperparameters. Table 2 shows that setting a proper value for the EMA decay
is significant. In all our experiments, the EMA decay is 0.99 during the first
37K iterations, and 0.999 afterward. The teacher benefits from new and
accurate student’s weight early in the training because the student improves
its segmentation capacity rapidly. On the other hand, since the student
improves slowly in the later training, the teacher can gain knowledge from the
old ensembled model.

The consistency loss weight $\delta$ follows the formula
$\delta=1+\delta_{0}e^{-5(1-x)^{2}}$, where $x\in[0,1]$ denotes the ratio
between the current epoch and the whole epochs and $\delta_{0}$ is a ramp-up
coefficient. Different from the usual sigmoid ramp-up [43], we add one to the
formula because it is essential to guarantee the contribution of the
consistency loss at the beginning of training. We decide to use $\delta_{0}$ =
30 for all our experiments.

###  5.4 Hyperparameter sensitivity on TGCF-DA

Figure 8: The change of augmented images $w.r.t$ the value of weight
$\lambda_{seg}$. From left to right: source input, output with $\lambda_{seg}$
= 1, output with $\lambda_{seg}$ = 10.

The weight $\lambda_{sem}$ for the semantic constraint is a hyperparameter for
training our augmentation network. Figure 8 shows some example results on
SYNTHIA $\rightarrow$ Cityscapes. When we use a lower value ($\lambda_{sem}$ =
1) for semantic constraint, the generator is prone to mix up objects and
scenes in the augmented images. On the other hand, the proper value for
semantic constraint ($\lambda_{sem}$ = 10) helps the network preserve the
local and global structures of images. These results confirm that the semantic
constraint enforces our augmentation network to retain semantic consistency.

##  6 Conclusion

We have proposed a novel framework comprised of two complementary approaches
for unsupervised domain adaptation for the semantic segmentation. We present
the GAN-based data augmentation with the guidance of target samples. Without
the use of cycle consistency, our augmentation network produces augmented
images for domain alignment. Moreover, the self-ensembling with those
augmented images can perform successful adaptation by transferring pseudo
labels from the teacher network to the student network. Experimental results
verify that our proposed model is superior to existing state-of-the-art
approaches.

Acknowledgements This work was supported by the National Research Foundation
of Korea (NRF) grant funded by the Korea government (MSIT)
(NRF-2018R1A5A7025409).

## References

  * [1] Amjad Almahairi, Sai Rajeshwar, Alessandro Sordoni, Philip Bachman, and Aaron Courville.  Augmented CycleGAN: Learning many-to-many mappings from unpaired data.  In Proceedings of the 35th International Conference on Machine Learning, pages 195–204, 2018.
  * [2] Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, and Andrew Gordon Wilson.  There are many consistent explanations of unlabeled data: Why you should average.  2018\.
  * [3] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Machine learning, 79(1-2):151–175, 2010.
  * [4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2018.
  * [5] Yuhua Chen, Wen Li, and Luc Van Gool.  Road: Reality oriented adaptation for semantic segmentation of urban scenes.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7892–7901, 2018.
  * [6] Yi-Hsin Chen, Wei-Yu Chen, Yu-Ting Chen, Bo-Cheng Tsai, Yu-Chiang Frank Wang, and Min Sun.  No more discrimination: Cross city adaptation of road scene segmenters.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1992–2001, 2017.
  * [7] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [8] Aysegul Dundar, Ming-Yu Liu, Ting-Chun Wang, John Zedlewski, and Jan Kautz.  Domain stylization: A strong, simple baseline for synthetic to real image domain adaptation.  arXiv preprint arXiv:1807.09384, 2018.
  * [9] Geoff French, Michal Mackiewicz, and Mark Fisher.  Self-ensembling for visual domain adaptation.  In International Conference on Learning Representations, 2018.
  * [10] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414–2423, 2016.
  * [11] Abel Gonzalez-Garcia, Joost van de Weijer, and Yoshua Bengio.  Image-to-image translation for cross-domain disentanglement.  In Advances in Neural Information Processing Systems, pages 1294–1305, 2018.
  * [12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [14] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  CyCADA: Cycle-consistent adversarial domain adaptation.  In Proceedings of the 35th International Conference on Machine Learning, pages 1989–1998, 2018.
  * [15] Judy Hoffman, Dequan Wang, Fisher Yu, and Trevor Darrell.  Fcns in the wild: Pixel-level adversarial and constraint-based adaptation.  arXiv preprint arXiv:1612.02649, 2016.
  * [16] Weixiang Hong, Zhenzhen Wang, Ming Yang, and Junsong Yuan.  Conditional generative adversarial network for structured domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1335–1344, 2018.
  * [17] Ehsan Hosseini-Asl, Yingbo Zhou, Caiming Xiong, and Richard Socher.  Augmented cyclic adversarial learning for low resource domain adaptation.  In International Conference on Learning Representations, 2019.
  * [18] Haoshuo Huang, Qixing Huang, and Philipp Krahenbuhl.  Domain transfer through deep activation matching.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 590–605, 2018.
  * [19] Sheng-Wei Huang, Che-Tsung Lin, Shu-Ping Chen, Yen-Yi Wu, Po-Hao Hsu, and Shang-Hong Lai.  Auggan: Cross domain adaptation with gan-based data augmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 718–731, 2018.
  * [20] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1501–1510, 2017.
  * [21] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal unsupervised image-to-image translation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 172–189, 2018.
  * [22] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [23] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim.  Learning to discover cross-domain relations with generative adversarial networks.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1857–1865. JMLR. org, 2017.
  * [24] Samuli Laine and Timo Aila.  Temporal ensembling for semi-supervised learning.  arXiv preprint arXiv:1610.02242, 2016.
  * [25] Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang.  Diverse image-to-image translation via disentangled representations.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 35–51, 2018.
  * [26] Chunyuan Li, Hao Liu, Changyou Chen, Yuchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin.  Alice: Towards understanding adversarial learning for joint distribution matching.  In Advances in Neural Information Processing Systems, pages 5495–5503, 2017.
  * [27] Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised image-to-image translation networks.  In Advances in Neural Information Processing Systems, pages 700–708, 2017.
  * [28] Shuangting Liu, Jiaqi Zhang, Yuxin Chen, Yifan Liu, Zengchang Qin, and Tao Wan.  Pixel level data augmentation for semantic image segmentation using generative adversarial networks.  arXiv preprint arXiv:1811.00174, 2018.
  * [29] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [30] Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala.  Deep photo style transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4990–4998, 2017.
  * [31] Liqian Ma, Xu Jia, Stamatios Georgoulis, Tinne Tuytelaars, and Luc Van Gool.  Exemplar guided unsupervised image-to-image translation with semantic consistency.  In International Conference on Learning Representations, 2019.
  * [32] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2794–2802, 2017.
  * [33] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In International Conference on Learning Representations, 2018.
  * [34] Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim.  Image to image translation for domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4500–4509, 2018.
  * [35] Christian S Perone, Pedro Ballester, Rodrigo C Barros, and Julien Cohen-Adad.  Unsupervised domain adaptation for medical imaging segmentation with self-ensembling.  arXiv preprint arXiv:1811.06042, 2018.
  * [36] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for data: Ground truth from computer games.  In European Conference on Computer Vision, pages 102–118. Springer, 2016.
  * [37] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3234–3243, 2016.
  * [38] Chuck Rosenberg, Martial Hebert, and Henry Schneiderman.  Semi-supervised self-training of object detection models.  In Proceedings of the Seventh IEEE Workshops on Application of Computer Vision (WACV/MOTION’05) - Volume 1 - Volume 01, pages 29–36. IEEE Computer Society, 2005.
  * [39] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3723–3732, 2018.
  * [40] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen.  Regularization with stochastic transformations and perturbations for deep semi-supervised learning.  In Advances in Neural Information Processing Systems, pages 1163–1171, 2016.
  * [41] Swami Sankaranarayanan, Yogesh Balaji, Arpit Jain, Ser Nam Lim, and Rama Chellappa.  Learning from synthetic data: Addressing domain shift for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3752–3761, 2018.
  * [42] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The Journal of Machine Learning Research, 15(1):1929–1958, 2014\.
  * [43] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In Advances in neural information processing systems, pages 1195–1204, 2017.
  * [44] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker.  Learning to adapt structured output space for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7472–7481, 2018.
  * [45] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  arXiv preprint arXiv:1607.08022, 2016.
  * [46] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8798–8807, 2018.
  * [47] Zuxuan Wu, Xintong Han, Yen-Liang Lin, Mustafa Gokhan Uzunbas, Tom Goldstein, Ser Nam Lim, and Larry S Davis.  Dcan: Dual channel-wise alignment networks for unsupervised scene adaptation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 518–534, 2018.
  * [48] Yonghao Xu, Bo Du, Lefei Zhang, Qian Zhang, Guoli Wang, and Liangpei Zhang.  Self-ensembling attention networks: Addressing domain shift for semantic segmentation.  In Thirty-Third AAAI Conference on Artificial Intelligence, 2019\.
  * [49] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong.  Dualgan: Unsupervised dual learning for image-to-image translation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2849–2857, 2017.
  * [50] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving video database with scalable annotation tooling.  arXiv preprint arXiv:1805.04687, 2018.
  * [51] Yang Zhang, Philip David, and Boqing Gong.  Curriculum domain adaptation for semantic segmentation of urban scenes.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2020–2030, 2017.
  * [52] Yiheng Zhang, Zhaofan Qiu, Ting Yao, Dong Liu, and Tao Mei.  Fully convolutional adaptation networks for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6810–6818, 2018.
  * [53] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2223–2232, 2017.
  * [54] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman.  Toward multimodal image-to-image translation.  In Advances in Neural Information Processing Systems, pages 465–476, 2017.
  * [55] Xinge Zhu, Hui Zhou, Ceyuan Yang, Jianping Shi, and Dahua Lin.  Penalizing top performers: Conservative loss for semantic segmentation adaptation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 568–583, 2018.
  * [56] Xiaojin Jerry Zhu.  Semi-supervised learning literature survey.  Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2005.
  * [57] Yang Zou, Zhiding Yu, BVK Vijaya Kumar, and Jinsong Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 289–305, 2018.


where $p_{\mathcal{I}}$ is the $p$-value for the intersection hypothesis
$\cap_{i\in\mathcal{I}}H^{-}_{i}$. Similarly, a valid $p$-value $q^{r/n}$ for
$K^{r/n}$ is

$q^{r/n}=\max_{\\{\mathcal{I}:\mathcal{I}\subseteq[n],|\mathcal{I}|=n-r+1\\}}q_{\mathcal{I}},$

where $q_{\mathcal{I}}$ is the $p$-value for the intersection hypothesis
$\cap_{i\in\mathcal{I}}K_{i}$.

For $f$ that satisfies the monotonicity and symmetry condition (A0),
$p^{r/n}=f(p_{(r)},\ldots,p_{(n)})$ is a valid $p$-value, satisfying

$\displaystyle\sup_{\theta\in H^{r/n}}\mathbb{P}_{\theta}(p^{r/n}\leq x)\leq
x\quad\forall x\in[0,1].$

The inequality is an equality, i.e., $p^{r/n}$ is uniformly distributed, for
the least favorable parameter configuration (LFC) $\theta_{LFC}\in H^{r/n}$,
for which the $p$-values corresponding to $r-1$ parameters are zero (almost
surely), and the $p$-values corresponding to the remaining $n-r+1$ parameters
are uniformly distributed. For example, in the normal means problem, the LFC
configuration for testing $H^{r/n}$ is that $r-1$ parameters are infinite and
$n-r+1$ parameters are zero; for testing $K^{r/n}$ the LFC is that $r-1$
parameters are minus infinity and $n-r+1$ parameters are zero.

For example, the PC $p$-values using Fisher’s combining method (Fisher,, 1934)
are:

$\displaystyle
p^{r/n}=\mathbb{P}\Big{(}\chi^{2}_{2(n-r+1)}\geq-2\sum_{k=r}^{n}\log(p_{(k)})\Big{)},\quad
q^{r/n}=\mathbb{P}\Big{(}\chi^{2}_{2(n-r+1)}\geq-2\sum_{k=r}^{n}\log(q_{(k)})\Big{)}$

where $p_{(1)}\leq\ldots\leq p_{(n)}$ and $q_{(1)}\leq\ldots\leq q_{(n)}$
denote the sorted values of $p_{1},\ldots,p_{n}$ and $q_{1},\ldots,q_{n}$,
respectively. Note that $1-p_{(k)}=q_{(n-k+1)}$ for continuous test
statistics.

By considering only hypotheses in $\mathcal{S}^{-}$ for $H^{r/n}$, we can
avoid including $p$-values that are stochastically much larger than uniform
when their null hypotheses are true. Therefore, as in § 3, we shall restrict
ourselves to the directions guided by the data, so we shall use for testing
$H^{r/n}$

$p^{r/n}=\max_{\\{\mathcal{I}:\mathcal{I}\subseteq[n],|\mathcal{I}|=n-r+1\\}}f(\\{2p_{i}:i\in\mathcal{I}\cap\mathcal{S}^{-}\\})=\begin{cases}f\left(2p_{(r)},\ldots,2p_{(|\mathcal{S}^{-}|)}\right)&\text{if
}r\leq|\mathcal{S}^{-}|,\\\ 1&\text{otherwise}.\end{cases}$ (24)

and for testing $K^{r/n}$

$q^{r/n}=\max_{\\{\mathcal{I}:\mathcal{I}\subseteq[n],|\mathcal{I}|=n-r+1\\}}f(\\{2q_{i}:i\in\mathcal{I}\cap\mathcal{S}^{+}\\})=\begin{cases}f\left(2q_{(r)},\ldots,2q_{(|\mathcal{S}^{+}|)}\right)&\text{if
}r\leq|\mathcal{S}^{+}|,\\\ 1&\text{otherwise}.\end{cases}$ (25)

######  Procedure E.1 (Adaptive PC testing).

  1. Step 1

Apply Step 1 of procedure 3.1.

  2. Step 2

Test in order $\\{H^{r/n}:r=1,\ldots,|\mathcal{S}^{-}|\\}$, using $p^{r/n}$ in
(24), at level $\alpha$. Stop at the first non-rejection, $p^{r/n}>\alpha$.
Let $l^{+}_{\alpha}$ be the number of rejections, with
$l^{+}_{\alpha}\in\\{0,\ldots,|\mathcal{S}^{-}|\\}$. If $l^{+}_{\alpha}=n$,
return $l^{+}_{\alpha}=n,l^{-}_{\alpha}=0$, otherwise go to the next step.

  3. Step 3

Test in order $\\{K^{r/n}:r=1,\ldots,n-|\mathcal{S}^{-}|\\}$, using $q^{r/n}$
in (25), at level $\alpha$. Stop at the first non-rejection, $q^{r/n}>\alpha$.
Let $l^{-}_{\alpha}$ be the number of rejections, with
$l^{-}_{\alpha}\in\\{0,\ldots,n-|\mathcal{S}^{-}|\\}$. Return $l^{+}_{\alpha}$
and $l^{-}_{\alpha}$ (with $l^{+}_{\alpha}+l^{-}_{\alpha}\leq n$).

Note that the bounds of the procedure will be the same if the testing in order
is continued until $n$ in each step. This is so because
$p^{(|\mathcal{S}^{-}|+1)/n}>\alpha$ and
$q^{(n-|\mathcal{S}^{-}|+1)/n}>\alpha$.

The guaranteed coverage is formalized in the following proposition.

######  Proposition E.1.

Let $\left\\{p^{r/n}:r\in[n]\right\\}$ be valid conditional PC $p$-values for
$\\{H^{r/n}:r\in[n]\\}$, and let $\left\\{q^{r/n}:r\in[n]\right\\}$ be valid
conditional PC $p$-values for $\\{K^{r/n}:r\in[n]\\}$. Then $l_{\alpha}^{+}$
and $l_{\alpha}^{-}$ satisfy (23). Furthermore, the unconditional coverage is

$\displaystyle\mathbb{P}_{\theta}\Big{(}l_{\alpha}^{+}\leq n^{+},\
l_{\alpha}^{-}\leq n^{-}+n^{0}\Big{)}\geq 1-(1-2^{-n})\alpha.$ (26)

###### Proof.

Suppose that $\theta$, the true parameter vector, has $n^{+}=t$ positive
coordinates. We have $l_{\alpha}^{+}\leq t\leq n-l_{\alpha}^{-}$ if and only
if $p^{(t+1)/n}>\alpha$ and $q^{(n-t+1)/n}>\alpha$. The result follows since
conditional on $\mathcal{S}^{-}$, it is only possible to make an error in one
direction. Specifically, if $t>|\mathcal{S}^{-}|$, then
$p^{(|\mathcal{S}^{-}|+1)/n}>\alpha$, since it is not possible to reject
$H^{(|\mathcal{S}^{-}|+1)/n}$, so it is not possible to err with regard to the
lower bound. Therefore, if $t>|\mathcal{S}^{-}|$, then

$\mathbb{P}_{\theta}(t\notin[l_{\alpha}^{+},n-l_{\alpha}^{-}]\mid\mathcal{S})=\mathbb{P}_{\theta}(t>n-l_{\alpha}^{-}\mid\mathcal{S})\leq\mathbb{P}_{\theta}(q^{(n-t+1)/n}\leq\alpha\mid\mathcal{S})\leq\alpha.$

Similarly, if $t<|\mathcal{S}^{-}|$, then
$q^{(n-|\mathcal{S}^{-}|+1)/n}>\alpha$, since it is not possible to reject
$K^{(n-|\mathcal{S}^{-}|+1)/n}$, so it is not possible to err with regard to
the upper bound. Therefore, if $t<|\mathcal{S}^{-}|$, then

$\mathbb{P}_{\theta}(t\notin[l_{\alpha}^{+},n-l_{\alpha}^{-}]\mid\mathcal{S})=\mathbb{P}_{\theta}(t<l_{\alpha}^{+}\mid\mathcal{S})\leq\mathbb{P}_{\theta}(p^{(t+1)/n}\leq\alpha\mid\mathcal{S})\leq\alpha.$

If $t=|\mathcal{S}^{-}|$, then since the lower bound is at most
$|\mathcal{S}^{-}|$ and the upper bound is at least $n-|\mathcal{S}^{+}|$, it
is not possible to make an error on either bound. Therefore, the unconditional
error of non-covering $n^{+}$ is

$\displaystyle\mathbb{P}_{\theta}(t\notin[l_{\alpha}^{+},n-l_{\alpha}^{-}])=\mathbb{E}\left(\mathbb{P}_{\theta}(t\notin[l_{\alpha}^{+},n-l_{\alpha}^{-}]\mid\mathcal{S})\right)$
$\displaystyle=\mathbb{E}_{\theta}\left\\{\mathbb{I}(t>|\mathcal{S}^{-}|)\mathbb{P}_{\theta}(t>n-l_{\alpha}^{-}\mid\mathcal{S}^{-})+\mathbb{I}(t<|\mathcal{S}^{-}|)\mathbb{P}_{\theta}(t<l_{\alpha}^{+}\mid\mathcal{S}^{-})\right\\}$
$\displaystyle\leq\alpha\mathbb{E}_{\theta}\left(\mathbb{I}(t>|\mathcal{S}^{-}|)+\mathbb{I}(t<|\mathcal{S}^{-}|)\right)=\alpha\mathbb{P}_{\theta}(t\neq|\mathcal{S}^{-}|)\leq\alpha(1-2^{-n}),$

where the last inequality follows from (19). ∎

######  Proposition E.2.

Algorithm 1 returns the lower bounds $\tilde{\ell}^{+}_{\alpha}$ and
$\tilde{\ell}^{-}_{\alpha}$. These lower bounds are at most as good as those
obtained by the Procedure E.1.

###### Proof.

It is sufficient to note that if $k<s$, then the index $v$ in (22) starts at
$\max(0,k-s)=0$ by computing the PC conditional $p$-value
$p^{k+1/n}=f(\\{2p_{(k+1)},\ldots,2p_{(s)}\\})$. Then $p^{k+1/n}>\alpha$
implies

$f_{k}=\max_{\mathcal{K}:|\mathcal{K}|=k}f(\\{2p_{i},i\in\mathcal{S}^{-}\cap\mathcal{K}^{c}\\}\cup\\{2q_{i},i\in\mathcal{S}^{+}\cap\mathcal{K}\\})\geq
p^{k+1/n}>\alpha$

for any $k\in\\{0,\ldots,s-1\\}$, i.e. the lower bound $\ell^{+}_{\alpha}$ for
$n^{+}$ from Algorithm 1 is smaller than or equal to the lower bound of
Procedure E.1. Likewise, if $k>s$ then the index $v$ in (22) starts at
$\max(0,k-s)=k-s$ by computing the PC conditional $p$-value
$q^{n-k+1/n}=f(\\{2q_{(n-k+1)},\ldots,2q_{(n-s)}\\})$, thus
$q^{n-k+1/n}>\alpha$ implies

$f_{k}=\max_{\mathcal{K}:|\mathcal{K}|=k}f(\\{2p_{i},i\in\mathcal{S}^{-}\cap\mathcal{K}^{c}\\}\cup\\{2q_{i},i\in\mathcal{S}^{+}\cap\mathcal{K}\\})\geq
q^{n-k+1/n}>\alpha$

for any $k\in\\{s+1,\ldots,n\\}$, i.e. the lower bound
$\tilde{\ell}^{-}_{\alpha}$ for $n^{-}+n_{0}$ from Algorithm 1 is less than or
equal to the upper bound of Procedure E.1. ∎

######  Remark E.1.

If $n^{0}>0$ then the probability that the lower bounds from Procedure E.1 do
not cover at least one of $n^{+},n^{-}$ may exceed $\alpha$. If $n^{0}=n$ and
all $p$ values are uniform then $\mathbb{P}(p^{1/n}\leq\alpha\cup
q^{1/n}\leq\alpha)\approx 2\alpha$; as $n^{0}$ decreases the probability that
the lower bounds do not cover at least one parameter decreases from $2\alpha$
to $\alpha$ (for $n_{0}=0$).

######  Remark E.2.

For a combination function $f()$, the test for qualitative interactions in
Zhao et al., (2019) is rejected at level $\alpha$ if and only if
$l_{\alpha}^{+}\geq 1$ and $l_{\alpha}^{-}\geq 1$ in the above procedure.
Therefore, if the assumption $n^{0}=0$ is reasonable, then the above procedure
complements nicely a conclusion that there is qualitative interaction, by
providing with $(1-\alpha)$ confidence the (interval) estimate of the
parameter tested, $n^{+}$. More generally, a level $\alpha$ test of a
generalized qualitative interaction null hypothesis that $n^{+}<a$ or
$n^{+}>b$, for predefined $1\leq a<b\leq n-1$, has the following rejection
rule: reject if $l_{\alpha}^{+}\geq a$ and $n-l_{\alpha}^{-}\leq b$. To see
that this is an $\alpha$ level test, consider the null value $\theta$ such
that $n^{+}(\theta)\notin[a,b]$. Without loss of generality, suppose
$n^{+}(\theta)>b$. Then the probability of falsely rejecting the generalized
qualitative interaction true null hypothesis is

$\mathbb{P}_{\theta}(l_{\alpha}^{+}\geq a\textrm{ and }n-l_{\alpha}^{-}\leq
b)\leq\mathbb{P}_{\theta}(n-l_{\alpha}^{-}\leq
b)\leq\mathbb{P}_{\theta}(n-l_{\alpha}^{-}<n^{+})\leq\alpha.$

###  E.1 A note on general confidence bounds for $n^{+}$

If $p^{r/n}$ is a valid $p$-value for testing $H^{r/n}$ for $r=1,\ldots,n$,
then

$\displaystyle l^{+}_{\alpha}(p)$ $\displaystyle=$
$\displaystyle\max\\{l\in\\{0,\ldots,n\\}:p^{r/n}\leq\alpha\,\,\mathrm{for}\,\,r=0,\ldots,l\\}$
(27)

satisfies $\mathbb{P}_{\theta}\big{(}l_{\alpha}(p)\leq n^{+}\big{)}\geq
1-\alpha$ for all $\theta\in\Theta$, where $p^{0/n}\equiv 0$ since
$H^{0/n}:n^{+}<0$ is always false. Analogously, if $q^{r/n}$ is a valid
$p$-value for testing $K^{r/n}$ for $r=1,\ldots,n$, then

$\displaystyle l_{\alpha}^{-}(p)$ $\displaystyle=$
$\displaystyle\max\\{u\in\\{0,\ldots,n\\}:q^{r/n}\leq\alpha\,\,\mathrm{for}\,\,r=0,\ldots,u\\}$
(28)

satisfies $\mathbb{P}_{\theta}\big{(}n^{+}\leq n-l^{-}_{\alpha}(p)\big{)}\geq
1-\alpha$ for all $\theta\in\Theta$, where $q^{0/n}\equiv 0$ since
$K^{0/n}:n^{+}>n$ is always false. For notational simplicity, we shall often
write $l^{+}_{\alpha}$ and $l^{-}_{\alpha}$ instead of $l^{+}_{\alpha}(p)$ and
$l^{-}_{\alpha}(p)$, but of course these bounds are functions of the $p$-value
vector $p$.

A straightforward application of the Bonferroni inequality shows that if level
$\alpha/2$ is used for each bound, i.e., $l^{+}_{\alpha/2}$ in (27) and
$l^{-}_{\alpha/2}$ in (28), then
$\mathbb{P}_{\theta}\big{(}l^{+}_{\alpha/2}\leq n^{+}\leq
n-l^{-}_{\alpha/2}\big{)}\geq 1-\alpha$. These bounds where used in Jaljuli et
al., (2022) in order to complement meta-analyses in systematic reviews.

The correction of using $\alpha/2$ in each direction (instead of $\alpha$, as
in the adaptive PC testing procedure ) is, however, conservative. Intuitively,
the correction should be less severe since in a given configuration, the
probability of erring by exceeding one bound is much larger than the
probability of erring by exceeding the other bound. To see this, note that

$\mathbb{P}_{\theta}(n^{+}\notin[l^{+}_{\alpha/2},n-l^{-}_{\alpha/2}]=\mathbb{P}_{\theta}(n^{+}<l^{+}_{\alpha/2})+\mathbb{P}_{\theta}(n^{+}>n-l^{-}_{\alpha/2})$

has value $\alpha/2$ for the following least favorable parameter
configurations (LFCs) when testing PC null hypotheses: the positive parameter
value has $p_{i}=0$ (almost surely) and the non-positive parameter value has a
$p$-value with a uniform distribution; or the positive parameter value has a
$p$-value which is (practically) uniformly distributed and the non-positive
parameter value has $q_{i}=0$ (almost surely). To see this, note that for the
LFC with $p_{i}=0$ for $n^{+}$ parameters, $p^{(n^{+}+1)/n}\sim U(0,1)$ and
$q^{(n-n^{+}+1)/n}=1$ almost surely. For the LFC with $q_{i}=0$ for $n-n^{+}$
parameters, $q^{(n-n^{+}+1)/n}\sim U(0,1)$ and $p^{(n^{+}+1)/n}=1$ almost
surely. Non-coverage can occur if the lower bound is violated, so
$p^{(n^{+}+1)/n}\leq\alpha/2$, or if the upper bound is violated, so
$q^{(n-n^{+}+1)/n}\leq\alpha/2$. Therefore

$\mathbb{P}_{LFC}(n^{+}\notin[l^{+}_{\alpha/2},n-l^{-}_{\alpha/2}])=\mathbb{P}(U\leq\alpha/2)=\alpha/2.$

We conjecture that the coverage guarantee is typically $(1-\alpha)$ if the
lower bounds are $l^{+}_{\alpha}$ and $l^{-}_{\alpha}$. In particular,
whenever the test statistics are continuous, from one dimensional exponential
families. Without loss of generality, suppose the first $t$ coordinates are
positive, i.e., $\theta_{1},\ldots,\theta_{t}>0$ and
$\theta_{t+1},\ldots,\theta_{n}\leq 0$, and $n^{+}=t$. Our conjecture is thus
that the solution to the following optimization problem is $\alpha$ for a
large class of valid PC $p$-values:

$\displaystyle\max_{\theta}$
$\displaystyle\mathbb{P}_{\theta}(p^{(t+1)/n}\leq\alpha)+\mathbb{P}_{\theta}(q^{(n-t+1)/n}\leq\alpha)$
s.t. $\displaystyle\theta_{i}>0,i=1,\ldots,t,$ $\displaystyle\theta_{j}\leq
0,j=t+1,\ldots,n.$

To see that this is not true in general for unconditional combination tests
(i.e., using local tests that do not condition on the vector of signs
$\mathcal{S}$), consider the following stylized example. Let $n=2$ and
$n^{+}=1$ be such that $\theta_{1}=0$ and $\theta_{2}$ is positive. Assume
that the distribution of a $p$-value from $H_{i}$, $x_{i}$, has the following
distribution: $\mathbb{P}(x_{i}=\alpha)=\alpha,\mathbb{P}(x_{i}=1)=1-\alpha.$
Similarly, the distribution of a $p$-value from $K_{i}$, $y_{i}=1-x_{i}$, has
distribution: $\mathbb{P}(y_{i}=\alpha)=\alpha,\mathbb{P}(y_{i}=1)=1-\alpha.$
These $p$-values are valid since

$\mathbb{P}_{H_{i}}(p_{i}\leq a)\leq a,\ \mathbb{P}_{K_{i}}(q_{i}\leq a)\leq
a,\ \forall a\in[0,1].$

The unconditional PC $p$-value (i.e., it is derived from a local test that
does not condition on the vector of signs $\mathcal{S}$) for $H^{2/2}$ and
$K^{2/2}$ is, respectively, $p^{2/2}=\max(p_{1},p_{2})$ which has distribution
$\max(1-x_{1},x_{2})$, and $q^{2/2}=\max(q_{1},q_{2})$ which has distribution
$\max(x_{1},1-x_{2})$. Therefore:

$\displaystyle\mathbb{P}_{\theta}(1\notin[l^{+}_{\alpha},n-l^{-}_{\alpha}]=\mathbb{P}_{\theta}(1<l^{+}_{\alpha})+\mathbb{P}_{\theta}(1<l^{-}_{\alpha})$
$\displaystyle=\mathbb{P}_{\theta}(\max(p^{1/2},p^{2/2})<\alpha)+\mathbb{P}_{\theta}(\max(q^{1/2},q^{2/2})<\alpha)$
$\displaystyle=\mathbb{P}((1-x_{1},x_{2})=(0,\alpha))+\mathbb{P}((x_{1},1-x_{2})=(\alpha,0))=2\times\alpha\times(1-\alpha)>\alpha.$

##  Appendix F Applications: enhancing meta-analysis

Figure 9 shows simultaneous $95\%$ confidence intervals for
$n^{+}(\mathcal{I}_{k})$ with Fisher’s combining function, where
$\mathcal{I}_{k}$ are the indexes of the top $k=|\mathcal{I}_{k}|$ schools
with largest (in absolute value) estimated effects $|y_{i}|$. For example,
among the top 38 schools, we observe at least 9 positive effects and at least
1 non-positive effect, or among the top 51 schools, we observe at least 10
positive effects and at least 2 non-positive effects, and so on.

$1$$56$056$k$$95\%$ confidence interval for $n^{+}(\mathcal{I}_{k})$ Figure 9:
Simultaneous $95\%$ confidence intervals for $n^{+}(\mathcal{I}_{k})$ with
Fisher’s combining function, where $\mathcal{I}_{k}$ are the indexes of the
top $k=|\mathcal{I}_{k}|$ schools with largest (in absolute value)
standardized effects. For example, within the top $38$ schools, we have at
least 9 positive effects and at least 1 non-positive effect (red interval); or
within the top $51$ schools, we have at least 10 positive effects and at least
2 non-positive effect (blue interval). Dots represent degenerate intervals
$[k,k]$.

##  Appendix G Additional Simulation results

For the simulation setting of $n=50$ parameters, described in § 6 of the main
text, we provide the following additional results.

First, for the FDR controlling procedure of Guo and Romano, (2015), GR-FDR, we
provide the actual coverage guarantee in the settings considered in Figure 10.
Specifically, we provide the estimated probability (based on 2000 simulation
runs for each data generation) that the number of positive and non-positive
discoveries does not exceed the true number of positive and non-positive
parameters, respectively, in the discovery set. The coverage can be much lower
than the 0.95 coverage guarantee, since this procedure only provides the
guarantee that the FDP is at most $\alpha$ in expectation. This is in contrast
to all other procedures that provide the 0.95 coverage guarantee.

0123$0.6$$0.65$$0.7$$0.75$$0.8$$0.85$$0.9$$0.95$$1$Signal-to-noise
ratioCoverage probability Figure 10:  For the GR-FDR procedure with discovery
set $\mathcal{R}$, the coverage probability of $n^{+}(\mathcal{R})$ and
$n^{-}(\mathcal{R})+n^{0}(\mathcal{R})$ versus the signal-to-noise ratio, for
the following settings: $n^{+}=n^{-}=15$ (black circles); $n^{+}=30$ and
$n^{-}=0$ (blue triangles). The horizontal red line is the desired coverage
probability of 0.95.

Second, for DCT and partitioning, we compare four combining methods: Fisher,
ALRT, Simes, and mSimes. Figure 11 shows that mSimes dominates Simes: it
provides tighter bounds and more discoveries. For the bounds, Fisher is best
and ALRT is a close second. For discoveries, this relation is reversed (with
ALRT, there are slightly more discoveries than with Fisher), but they are both
much worse than Simes.

Finally, we note that we performed additional simulations, omitted for
brevity, varying the parameter values within each data generation settings and
considering more configurations of $n^{+}$ and $n^{-}$. The qualitative
conclusions above regarding the relative performance of the different
combining methods remained unchanged.

0123$0$$5$$10$$15$$20$$25$Signal-to-noise ratio$\ell^{+}+\ell^{-}$AP FisherAP
ALRTAP SimesAP mSimesDCT FisherDCT ALRTDCT SimesDCT mSimes

0123$0$$5$$10$$15$$20$$25$Signal-to-noise
ratio$|\mathcal{D}^{+}|+|\mathcal{D}^{-}|$FisherALRTSimesmSimes

0123$0$$5$$10$$15$$20$$25$Signal-to-noise ratio$\ell^{+}+\ell^{-}$AP FisherAP
ALRTAP SimesAP mSimesDCT FisherDCT ALRTDCT SimesDCT mSimes

0123$0$$5$$10$$15$$20$$25$Signal-to-noise
ratio$|\mathcal{D}^{+}|+|\mathcal{D}^{-}|$FisherALRTSimesmSimes

Figure 11:  The average sum of the lower bounds on positive and nonpositive
parameters (left column) and number of individual hypotheses rejected (right
column), versus the signal-to-noise ratio, for the following methods: AP with
combining functions Fisher (AP FISHER), ALRT (AP ALRT), Simes (AP SIMES),
modified Simes (AP mSIMES); DCT with combining functions Fisher (DCT FISHER),
ALRT (DCT ALRT), Simes (DCT SIMES), modified Simes (DCT mSIMES);.
$n^{+}=n^{-}=15$ in the first row; $n^{+}=30$ and $n^{-}=0$ in the second row.
In the right column, the absence of DCT procedures is due to the fact that
they coincide with the AP procedures.


11institutetext: Department of Electrical and Computer Engineering, University
of Pittsburgh, Pittsburgh, PA, USA 11email<EMAIL_ADDRESS>22institutetext:
Perelman School of Medicine, Philadelphia, University of Pennsylvania, PA, USA
33institutetext: Keck School of Medicine, University of Southern California,
Marina del Rey, CA, USA

# Incomplete Multimodal Learning for Complex Brain Disorders Prediction

Reza Shirkavand 11 Liang Zhan 11 Heng Huang 11 Li Shen 22 Paul M. Thompson 33

###### Abstract

Recent advancements in the acquisition of various brain data sources have
created new opportunities for integrating multimodal brain data to assist in
early detection of complex brain disorders. However, current data integration
approaches typically need a complete set of biomedical data modalities, which
may not always be feasible, as some modalities are only available in large-
scale research cohorts and are prohibitive to collect in routine clinical
practice. Especially in studies of brain diseases, research cohorts may
include both neuroimaging data and genetic data, but for practical clinical
diagnosis, we often need to make disease predictions only based on
neuroimages. As a result, it is desired to design machine learning models
which can use all available data (different data could provide complementary
information) during training but conduct inference using only the most common
data modality. We propose a new incomplete multimodal data integration
approach that employs transformers and generative adversarial networks to
effectively exploit auxiliary modalities available during training in order to
improve the performance of a unimodal model at inference. We apply our new
method to predict cognitive degeneration and disease outcomes using the
multimodal imaging genetic data from Alzheimer’s Disease Neuroimaging
Initiative (ADNI) cohort. Experimental results demonstrate that our approach
outperforms the related machine learning and deep learning methods by a
significant margin.

###### Keywords:

Incomplete Multimodal Learning Brain Cognitive Prediction Brain Disease
Transformers

##  1 Introduction

Alzheimer’s disease (AD) is a progressive neurodegenerative disease with
primary symptoms of memory loss and other cognitive deficits. Despite decades
of studies on AD, effective treatments are only just emerging [5, 8].
Consequently, early diagnosis is extremely important for planning support
strategies and improving the quality of patients’ lives.

AD symptoms are usually reflected by the cognitive status. People diagnosed as
having mild cognitive impairment (MCI) are at a higher risk of progressing to
AD [16]. Several studies [29, 24, 20] have been conducted to predict the
progression from MCI to AD and aim to identify predictors that can distinguish
stable MCI (sMCI) from progressive or converting MCI (pMCI or cMCI). Cognitive
measures that carry information about cognitive status can help in diagnosing
people at risk of AD in the earlier stages of the disease.

Many recent studies suggested that combining multiple data modalities like
structure MRI, PET, genetic data, or CSF biomarkers can improve diagnostic and
prognostic accuracy for AD at different stages. Multimodal learning (MML)
outperforms unimodal methods [14, 26], but the acquisition cost of some
modalities, particularly genetic data, limits the application of general MML
models in AD studies and other medical problems. Exploiting multiple
modalities in training, but only using one principal modality, _e.g._ MRI -
which is the most common neuroimaging modality for studying AD - at inference
would be a practical workaround. Consequently, there is a strong motivation to
develop incomplete multimodal learning methods that can deal with missing or
partial data modalities to boost prediction performance.

Over the past decade, deep learning has been successfully used to tackle
various problems in image, audio, and text domains [10]. This tremendous
success has been due to the inductive biases of deep neural networks taking
advantage of spatial, temporal, or semantic structures in the data. However,
in the absence of such structure in tabular data (biomedical data are often
processed with domain knowledge to extract features, such as the summary
statistics values extracted from brain structure MRI data, instead of using
the whole images) common in the biomedical and bio-medicine fields, deep
neural networks have been less effective than their non-deep machine learning
competitors, Gradient Boosting Decision Trees (GBDTs) [6, 18, 23].
Nevertheless, the application of deep learning to such data is still an
appealing solution [3, 4, 12, 19, 22] as it would enable the formation of
multimodal pipelines that use additional data modalities correlated with the
downstream task resulting in potentially superior performance over GBDTs.

We propose a novel incomplete multimodal learning method to predict the
patients’ brain disease outcomes and their cognitive status using the
multimodal imaging genetics data. We train a single-modality model on MRI data
and a multimodal learning model using both MRI and genetic data
collaboratively. As the backbone structure of our model, we adopt an
adaptation of the transformer architecture [12]. We use a generative
adversarial network (GAN) [11] to guide the single-modal model to imitate the
auxiliary SNP modality and enhance the main MRI modality. The empirical
studies on brain imaging genetic data show our new framework improves the
representational power of the model and significantly outperforms existing
baselines.

##  2 Method

###  2.1 Problem Formulation and Our Motivations

In our supervised setting, we are given a dataset
$D=\\{x_{i},y_{i}\\}_{i=1}^{N}$, where $x_{i}=(x_{i}^{GEN},x_{i}^{MRI})\in X$
represents the genetic modality $x_{i}^{GEN}$ and the MRI features
$x_{i}^{MRI}$ of each subject, and $y_{i}$ is the corresponding continuous
subject’s cognitive scores ($y_{i}\in R$) or discrete disease outcome,
($y_{i}\in\\{CN,MCI,AD\\}$).

Fig. 1 illustrates our proposed incomplete multimodal learning framework.
During training, we pre-train a multimodal teacher model $M$ on both MRI and
genetic data, which has separate backbones for each data source. A final
regressor combines the extracted features to produce the output. During
testing, we only have access to MRI data. Our goal is to improve the
performance of the uni-modal model $U$ that uses MRI-extracted features
$\\{x_{i}^{MRI}\\}_{i=1}^{N}$ from brain ROI volumes to predict $y_{i}$. The
model $U$ includes a GAN component that mimics the genetic component of the
model $M$ using MRI data only. We use a transformer network as the backbone
for both models $U$ and $M$. Our method substantially outperforms existing
state-of-the-art GBDTs and DL networks.

Figure 1: Overview of our incomplete multimodal framework. Top Left: The
$M$-Model. Neuroimaging features (measurements from 260 brain ROI volumes) and
SNP data are fed into the corresponding MRI transformer component and the
genetic backbone of the $M$-Model. The resulting representations are
concatenated and used as the final regressor input. Top Right: The transformer
architecture. First, 1-D features are mapped to an embedding space. Then
several transformer blocks are applied. Bottom: Training of the $U$-Model. We
guide the $U$-Model to imitate the genetic representations space of the
$M$-Model using a Riemannian GAN. The MRI transformer components of the
$M$-Model and the $U$-Model are trained collaboratively.

###  2.2 The $M$-Model

The multimodal teacher, i.e. the $M$ Model, has two separate sub-nets
designated for MRI features and SNP modalities. The choice of the subnet
architecture is independent of the proposed method. Upon experiments, we show
that the use of transformers outperforms other choices of MRI backbone
architecture, e.g. MLP. We use a similar Transformer architecture to [12] as
the backbone. This component first takes its input $x$ to an embedding space
$h$ via an affine map

$h_{j}^{mod}=x_{j}^{mod}.W_{j}^{mod}+b_{j}^{mod}$ (1)

with $x_{j}^{mod}$ being the $j$-th feature of a particular modality and
$W_{j}^{mod},b_{j}^{mod}\in R^{d}$ being weights and biases, respectively. The
final embedding $h^{mod}$ is the simple concatenation of all embeddings of a
modality:

$h^{mod}=Concat[h_{1}^{mod},\cdots,h_{d}^{mod}]\,.$ (2)

Then, $L$ Transformer blocks $f_{\theta}^{k}$ are applied to the embeddings to
produce the final representation of a modality $z^{mod}$:

$z^{mod}=f_{\theta}^{L}(\cdots f_{\theta}^{1}(h^{mod}))\,.$ (3)

Finally, a linear regression or classification head $g$ acts on the
concatenation of the derived features from each modality to produce the
$M$-Model output $\hat{y}$ via:

$\hat{y}=g(Concat[z^{MRI},z^{GEN}])\,.$ (4)

Fig. 1 shows the $M$ model architecture as well as the architecture of the
transformer backbone. The model is trained with the Mean Squared Error(MSE)
loss.

###  2.3 The $U$-Model

The $U$-model is used for inference and shares the same MRI backbone as the
$M$-model (refer to Fig. 1). To enhance the $U$-model’s prediction, we include
a Riemannian GAN component that mimics the representation space of the genetic
component of the $M$-model. By imitating the lower-dimensional representation
space, our method is more efficient than attempting to estimate the entire
genetic data from MRI features.

The discriminator in Riemannian GANs is different from conventional GAN
models, which use Euclidean distance to distinguish between real and fake
samples. Instead, the discriminator in Riemannian GANs measures the length of
geodesics of a low-dimensional representation of the samples on a Riemannian
manifold, such as a hypersphere. This approach provides greater stability
during training because distances on a hyper-sphere are bounded, unlike
Euclidean distance in the original sample space [25].

To implement our Riemannian GAN, we use a trainable $d$-sphere, $S$, with a
center, $c$, and a main axis, $v$, both in $R^{d}$. The discriminator takes a
genetic representation, $h\in R^{d_{2}}$, whether real or fake, and maps it to
an embedding, $h^{\prime}\in R^{d}$, with $d<d_{2}$. It then projects
$h^{\prime}$ onto the $d$-sphere using
$proj_{S}\mathbf{h^{i}}=\frac{h^{\prime}-c}{||h^{\prime}-c||_{2}}$. The
discriminator assigns higher scores to samples that project onto larger radii
cross-sections, with normal vector $v$. More formally, we decompose the
projection of a sample $h^{i}_{proj}$ into two perpendicular components,
$proj_{S}\mathbf{h^{i}}=proj_{v}\mathbf{h^{i}}+proj_{v^{\perp}}\mathbf{h^{i}}$.
The discriminator then outputs the score:

$D(h^{i})=\frac{||proj_{v^{\perp}}\mathbf{h^{i}}||_{2}}{\sigma[proj_{v^{\perp}}\mathbf{h^{i}}]}-\frac{||proj_{v}\mathbf{h^{i}}||_{2}}{\sigma[proj_{v}\mathbf{h^{i}}]}\,,$
(5)

where $\sigma$ is the empirical standard deviation calculated over a batch of
real or fake samples of size $B$, and is defined as
$\sigma(h)=\sum_{i=1}^{B}\frac{||h^{i}||_{2}}{B}$.

In order to enhance stability during training and mitigate the impact of
outliers and mode collapse, the GAN discriminator employs the relativistic
averaged loss approach [17], using the following objective function:

$\begin{split}L_{GAN}^{D}&=-E_{h\sim
p_{\text{GEN}}}[log(\text{sigmoid}(\eta(D(h)-E_{h^{\prime}\sim
p_{\text{U-pred}}}[D(h^{\prime})])))]\\\ &-E_{h^{\prime}\sim
p_{\text{U-pred}}}[log(\text{sigmoid}(\eta(E_{h\sim
p_{\text{GEN}}}[D(h)]-D(h^{\prime}))))]\end{split}$ (6)

Here, $p_{GEN}$ represents the distribution of the real samples in the genetic
representation space of the M-model, $p_{U-pred}$ represents the distribution
of fake samples generated by the generator module of the U-model, and $\eta$
is a hyper-parameter that adjusts the range of the score difference of the
sigmoid function [17].

The generator aims to generate samples that are projected near the largest
hyper-section of the $d$-sphere, with an adversarial loss $L_{GAN}^{G}$ that
is the inverse of the discriminator loss. The center of the $d$-sphere is
identified as the point that minimizes the sum of squared distances to all
samples, which is optimized using the Huber loss function [15]. This function
offers a balance between mean squared error and mean absolute error, making it
more resilient to outliers and suitable for a wider range of data
distributions. Given a batch of samples $B=\\{h^{i}\\}$, we compute the
following objective:

$L_{c}=\frac{1}{|B|}\sum_{i=1}^{|B|}L_{Huber}(||h_{proj}^{i}-c||_{2})$ (7)

Here, $L_{Huber}$ is defined as:

$L_{Huber}=\begin{cases}0.5x^{2}&\text{if }x\leq 1\\\
x-0.5&\text{otherwise}\end{cases}$ (8)

After fixing the center, we minimize the following objective to force the
discriminator to project all samples to points with similar distances to the
center and find the main axis $v$:

$L_{distance}=\frac{1}{|B|}\sum_{i=1}^{|B|}L_{Huber}(||h_{proj}^{i}-c||_{2}-\sigma[h^{i}_{proj}])$
(9)

We guide the U-Model to replicate the M-model representation of the genetic
modality using the the Kullback-Leibler divergence between the output
distributions of the U-Model and M-Model:

$L_{im}=KL(P_{U}(y|x;T),P_{M}(y|x;T))$ (10)

The temperature parameter $T$ controls the distribution’s sharpness. The
U-Model’s total loss is a weighted combination of the imitation loss $L_{im}$,
the Mean Squared Error (MSE) loss $L_{MSE}$, and the U-Model generator loss
$L_{GAN}^{G}$:

$L_{U}=L_{MSE}+\alpha L_{GAN}+\beta L_{im}$ (11)

The GAN loss is excluded from updating the backbone transformer’s parameters
due to its instability. We use only the MSE loss together with the imitation
loss for that purpose. Also, the M-Model’s MRI transformer parameters are
updated using an exponential moving average scheme [28] to match the
corresponding parameters in the U-Model. The update rule is expressed as:

$\theta_{M-MRI}=\gamma\theta_{M-MRI}+(1-\gamma)\theta_{U-MRI}$ (12)

##  3 Experiments

The goal of our experiments is to identify how well our proposed method can
predict cognitive scores and disease outcome from MRI-extracted features. We
conduct experiments on our dataset and compare the proposed method with the
state-of-the-art traditional machine learning and deep learning models that
are suited to our data in general.

###  3.1 Experimental Setup

####  3.1.1 Data Description

In our experiments, we use the imaginge genetic data from the Alzheimer’s
Disease Neuroimaging Initiative (ADNI) cohort [1], which collects various
datasets and biomarkers for Alzheimer’s disease studies. The image features
are summary statistics extracted from the structural MRIs of 844 subjects
based on 260 brain ROIs [7].

The original ADNI cohort provides the raw cognitive scores for each patients.
More recently a collaborative study by ADSP Phenotype Harmonization Consortium
[2] conducted harmonization for the memory scores to make them comparable
across different data collection institutes. Four harmonized cognitive
parameters are finally provided to summarize the raw memory score sets such as
MMSI, Fluency, MoCA, Trails A/B, _etc_. The harmonized cognitive parameters
include four categories: the memory score (PHC_MEM), the executive function
score (PHC_EXF), the language score (PHC_LAN), and the visuospatial score
(PHS_VSP). However, PHS_VSP was removed from the analysis due to a large
number of missing values.

Genetic data in our experiments are the Single Nucleotide Polymorphisms (SNP).
The SNP data and all ADNI datasets had 2,001 subjects and 76K SNPs in common,
with only 775 shared among all three modalities (MRI - SNP - Cognitive). These
subjects comprised 175 Alzheimer’s Dementia (AD), 380 Mild Cognitive
Impairment (MCI), and 220 Cognitively Normal (CN) subjects. To prevent
overfitting, we selected only the SNPs known to be related to AD, which were
identified from a list of 287 genes with their SNPs published on the NCBI
website [21]. After filtering out irrelevant SNPs, around 5000 SNPs were kept.

####  3.1.2 Genetic Data Pre-processing

Quality Control of the SNP data consists of three main steps:
removing/imputing the NA values, filtering of the SNP based on Minor Allele
Frequency (MAF), and Hardy-Weinberg Equation (HWE). For this step of quality
control, we removed SNPs with more than 95% missing values and imputed the
rest with the expected value as the round mean. We also filtered all the SNPs
with MAF less than 0.05 and Hardy-Weinberg p-values smaller than $1\times
10^{-6}$. After this quality control, 1079 SNPs remained.

####  3.1.3 Evaluation Metric and Training/Testing Settings

We evaluate the effectiveness of our method in regressing cognitive scores
using Root Mean Squared Loss (RMSE) and the coefficient of determination
$r^{2}$. Moreover, we report macro-averaged accuracy, precision, recall and f1
score as our comparison metric for the patient disease classification task. We
carry out 5-fold cross-validation (5-fold CV) to make a fair comparison of our
method with the baselines. We use the Adam optimizer and L-2 regularization in
training. more details about the training settings can be found on our GitHub
repository when the paper is published.

####  3.1.4 Baselines

We compare with existing work in the space of structured data prediction as
well as ablations that test components of our framework. Our primary points of
comparison are Gradient Boosting Decision Trees (GBDT) [9] variants, namely
XGBoost [6], LightGBM [18], and CatBoost [23]. However, we also evaluate other
recently proposed deep learning methods Neural Oblivious Decision Ensembles
(NODE) [22], GrowNet [4], Self-Normalizing Networks (SNN) [19] as well as
simple yet powerful deep architectures Transformers [27] and ResNet [13].

###  3.2 Experimental Results

Table 1 and Table 2 show the performance of baseline models as well as our
incomplete multimodal learning framework with transformer backbone on the
dataset on two complex brain disorders prediction tasks: predicting cognitive
scores and disease outcomes. The results demonstrate the consistent and
substantial superiority of our method over the existing state of the art
approaches. It must be noted that the primary point of reference of the
incomplete multi-modal learning approach is the U model. This model only has
access to the MRI modality. Hence the comparison with other single-modality
baselines is fair.

Our method offers a twofold superiority. Firstly, the two modalities share
some information that is correlated with the outcomes, and the generative
adversarial network (GAN) can effectively leverage the MRI data to reconstruct
the representation space of the genetic data, thereby enhancing the model’s
performance. Secondly, the transformer network further improves the model’s
representation power by identifying inter-variable dependencies that other
machine learning and deep learning methods cannot detect. This results in a
more accurate and robust prediction model.

Table 1: Comparison of performance of baselines with our proposed method.
Values correspond to the mean and standard deviation of root mean squared
error and $r^{2}$ regression coefficient across 5-fold cross-validation of
regressing cognitive scores from MRI data.

Method | PHC_MEM | PHC_EXF | PHC_LAN
---|---|---|---
RMSE | $r^{2}$ | RMSE | $r^{2}$ | RMSE | $r^{2}$
XGBoost [6] | 0.391 ± 0.022 | 0.337 ± 0.071 | 0.442 ± 0.012 | 0.080 ± 0.073 | 0.385 ± 0.024 | 0.192 ± 0.058
Light GBM [18] | 0.377 ± 0.021 | 0.381 ± 0.048 | 0.419 ± 0.014 | 0.174 ± 0.070 | 0.385 ± 0.022 | 0.191 ± 0.041
Catboost [23] | 0.368 ± 0.021 | 0.413 ± 0.051 | 0.408 ± 0.012 | 0.216 ± 0.062 | 0.372 ± 0.021 | 0.245 ± 0.045
MLP | 0.367 ± 0.020 | 0.433 ± 0.052 | 0.376 ± 0.012 | 0.249 ± 0.057 | 0.327 ± 0.013 | 0.268 ± 0.080
Resnet [13] | 0.365 ± 0.013 | 0.395 ± 0.056 | 0.393 ± 0.016 | 0.214 ± 0.082 | 0.320 ± 0.015 | 0.271 ± 0.051
SNN [19] | 0.433 ± 0.018 | 0.358 ± 0.063 | 0.419 ± 0.029 | 0.349 ± 0.074 | 0.372 ± 0.036 | 0.252 ± 0.069
Node [22] | 0.404 ± 0.033 | 0.388 ± 0.061 | 0.390 ± 0.027 | 0.237 ± 0.078 | 0.332 ± 0.022 | 0.259 ± 0.069
Our Single-Modality U-Model | 0.200 ± 0.090 | 0.555 ± 0.051 | 0.283 ± 0.065 | 0.446 ± 0.062 | 0.280 ± 0.041 | 0.301 ± 0.044

Table 2: Comparison of performance of baselines with our proposed method.
Values correspond to the mean and standard deviation of each classification
metric across 5-fold cross-validation of classifying disease progression
(CN/MCI/AD) from MRI data.

Method |  Accuracy |  Precision |  Recall |  F1-score
---|---|---|---|---
XGboost[6] |  0.790 ± 0.022 |  0.801 ± 0.023 |  0.790 ± 0.022 |  0.794 ± 0.023
Light GBM[18] |  0.808 ± 0.026 |  0.821 ± 0.019 |  0.808 ± 0.026 |  0.812 ± 0.024
Catboost[23] |  0.843 ± 0.031 |  0.853 ± 0.019 |  0.843 ± 0.031 |  0.846 ± 0.033
MLP |  0.837 ± 0.017 |  0.835 ± 0.019 |  0.837 ± 0.017 |  0.835 ± 0.017
Resnet[13] |  0.853± 0.025 |  0.846 ± 0.023 |  0.853 ± 0.025 |  0.848 ± 0.024
SNN[19] |  0.851 ± 0.031 |  0.860 ± 0.043 |  0.850 ± 0.031 |  0.852 ± 0.037
Node[22] |  0.818 ± 0.027 |  0.822 ± 0.023 |  0.818 ± 0.027 |  0.818± 0.024
Ours |  0.927 ± 0.026 |  0.925 ± 0.028 |  0.927 ± 0.026 |  0.925 ± 0.027

###  3.3 Ablation Studies

We performed ablations on our proposed framework to evaluate the impact of
using incomplete multimodal learning compared to a vanilla transformer.
Furthermore, we examined the effectiveness of the transformer component by
comparing our approach’s results to an MLP backbone. Table 3 summarizes our
findings, which indicate that our proposed method, in combination with the
transformer backbone, yields superior results in all unimodal settings. In
addition, we present the M-model’s performance, which leverages both
modalities for prediction and serves as an upper bound for other values.

Table 3: Comparison of the proposed method with ablations. Values correspond
to the mean and standard deviation of root mean squared error across 5-fold
cross-validation.

Method |  PHC_MEM |  PHC_EXF |  PHC_LAN
---|---|---|---
Vanilla Transformer [12] |  0.372 ± 0.028 |  0.361 ± 0.019 |  0.318 ± 0.016
MLP Backbone |  0.323 ± 0.026 |  0.359 ± 0.026 |  0.315 ± 0.016
M-Model (Both Modalities) |  0.171 ± 0.025 |  0.247 ± 0.022 |  0.257 ± 0.022
U-Model |  0.200 ± 0.090 |  0.283 ± 0.065 |  0.280 ± 0.041

##  4 Conclusion

This study introduced a novel approach to incomplete multimodal learning that
leverages both neuroimaging features and genetic data to predict cognitive
conditions in the brain. We validated the performance of our proposed method
using ADNI imaging data and SNP genetics data and found it to outperform
existing baselines. Beyond biomedical applications, our approach can be
utilized in tasks where there is a primary modality and supplementary but
expensive sources of data that may be available during training but not at
inference.

## References

  * [1] ADNI: Alzheimer’s disease neuroimaging initiative, http://adni.loni.usc.edu/, accessed: Nov 3, 2022
  * [2] ADSP: Alzheimer’s disease sequencing project, https://adsp.niagads.org/, accessed: Oct 18, 2022
  * [3] Arik, S.Ö., Pfister, T.: Tabnet: Attentive interpretable tabular learning. In: Proc. of the AAAI Conf. on AI. vol. 35, pp. 6679–6687 (2021)
  * [4] Badirli, S., Liu, X., Xing, Z., Bhowmik, A., Doan, K., Keerthi, S.S.: Gradient boosting neural networks: Grownet. arXiv preprint arXiv:2002.07971 (2020)
  * [5] Barthélemy, N.R., Li, Y., Joseph-Mathurin, N., Gordon, B.A., Hassenstab, J., Benzinger, T., Buckles, V., Fagan, A.M., Perrin, R.J., Goate, A.M., et al.: A soluble phosphorylated tau signature links tau, amyloid and the evolution of stages of dominantly inherited alzheimer’s disease. Nature medicine 26(3), 398–407 (2020)
  * [6] Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proc. of the 22nd acm SIGKDD. pp. 785–794 (2016)
  * [7] Davatzikos, C.: Hierarchical parcellation of mri using multi-atlas labeling methods (2014), https://www.rad.upenn.edu/sbia/
  * [8] van Dyck, C.H., Swanson, C.J., Aisen, P., Bateman, R.J., Chen, C., Gee, M., Kanekiyo, M., Li, D., Reyderman, L., Cohen, S., et al.: Lecanemab in early alzheimer’s disease. New England Journal of Medicine (2022)
  * [9] Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Annals of statistics pp. 1189–1232 (2001)
  * [10] Goodfellow, I., Bengio, Y., Courville, A.: Deep learning. MIT press (2016)
  * [11] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Communications of the ACM 63(11), 139–144 (2020)
  * [12] Gorishniy, Y., Rubachev, I., Khrulkov, V., Babenko, A.: Revisiting deep learning models for tabular data. Advances in NeurIPS 34, 18932–18943 (2021)
  * [13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proc. of the IEEE CVPR. pp. 770–778 (2016)
  * [14] Huang, Y., Du, C., Xue, Z., Chen, X., Zhao, H., Huang, L.: What makes multi-modal learning better than single (provably). Advances in NeurIPS 34, 10944–10956 (2021)
  * [15] Huber, P.J.: Robust estimation of a location parameter. In: Breakthroughs in statistics, pp. 492–518. Springer (1992)
  * [16] Jack Jr, C.R., Bennett, D.A., Blennow, K., Carrillo, M.C., Dunn, B., Haeberlein, S.B., Holtzman, D.M., Jagust, W., Jessen, F., Karlawish, J., et al.: Nia-aa research framework: toward a biological definition of alzheimer’s disease. Alzheimer’s & Dementia 14(4), 535–562 (2018)
  * [17] Jolicoeur-Martineau, A.: The relativistic discriminator: a key element missing from standard gan. arXiv preprint arXiv:1807.00734 (2018)
  * [18] Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.Y.: Lightgbm: A highly efficient gradient boosting decision tree. Advances in NeurIPS 30 (2017)
  * [19] Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. Advances in NeurIPS 30 (2017)
  * [20] Li, H.T., Yuan, S.X., Wu, J.S., Gu, Y., Sun, X.: Predicting conversion from mci to ad combining multi-modality data and based on molecular subtype. Brain sciences 11(6), 674 (2021)
  * [21] NCBI: National library of medicine, national center for biotechnology information, https://www.ncbi.nlm.nih.gov/, accessed: Oct 15, 2022
  * [22] Popov, S., Morozov, S., Babenko, A.: Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312 (2019)
  * [23] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., Gulin, A.: Catboost: unbiased boosting with categorical features. Advances in NeurIPS 31 (2018)
  * [24] Schmidt-Richberg, A., Guerrero, R., Ledig, C., Molina-Abril, H., Frangi, A.F., Rueckert, D., Initiative, A.D.N., et al.: Multi-stage biomarker models for progression estimation in alzheimer’s disease. In: IPMI. pp. 387–398. Springer (2015)
  * [25] Shim, W., Cho, M.: Circlegan: generative adversarial learning across spherical circles. Advances in NeurIPS 33, 21081–21091 (2020)
  * [26] Sun, X., Xu, Y., Cao, P., Kong, Y., Hu, L., Zhang, S., Wang, Y.: Tcgm: An information-theoretic framework for semi-supervised multi-modality learning. In: ECCV. pp. 171–188. Springer (2020)
  * [27] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in NeurIPS 30 (2017)
  * [28] Zhang, Y., Xiang, T., Hospedales, T.M., Lu, H.: Deep mutual learning. In: Proc. of the IEEE CVPR. pp. 4320–4328 (2018)
  * [29] Zhu, Y., Zhu, X., Kim, M., Shen, D., Wu, G.: Early diagnosis of alzheimer’s disease by joint feature selection and classification on temporally structured support vector machine. In: MICCAI. pp. 264–272. Springer (2016)