# Deconvolving Complex Neuronal Networks into Interpretable Task-Specific
Connectomes

Yifan Wang, Ananth Grama
Department of Computer Science,
Purdue University,
West Lafayette, IN, USA
{wang5617<EMAIL_ADDRESS>
&Vikram Ravindra
Department of Computer Science,
University of Cincinnati,
Cincinnati, OH, USA
<EMAIL_ADDRESS>

###### Abstract

Task-specific functional MRI (fMRI) images provide excellent modalities for
studying the neuronal basis of cognitive processes. We use fMRI data to
formulate and solve the problem of deconvolving task-specific aggregate
neuronal networks into a set of basic building blocks called canonical
networks, to use these networks for functional characterization, and to
characterize the physiological basis of these responses by mapping them to
regions of the brain. Our results show excellent task-specificity of canonical
networks, i.e., the expression of a small number of canonical networks can be
used to accurately predict tasks; generalizability across cohorts, i.e.,
canonical networks are conserved across diverse populations, studies, and
acquisition protocols; and that canonical networks have strong anatomical and
physiological basis. From a methods perspective, the problem of identifying
these canonical networks poses challenges rooted in the high dimensionality,
small sample size, acquisition variability, and noise. Our deconvolution
technique is based on non-negative matrix factorization (NMF) that identifies
canonical networks as factors of a suitably constructed matrix. We demonstrate
that our method scales to large datasets, yields stable and accurate factors,
and is robust to noise.

_K_ eywords deconvolving connectomes, matrix factorization, dimension
reduction, functional MRI.

##  1 Introduction

Connectomic studies use functional brain images of subjects performing tasks
to understand complex cognitive processes. Functional Magnetic Resonance
Imaging (fMRI) is a common imaging modality used to analyze the underlying
natural processes in healthy individuals and the dysregulation of such
processes due to disease and/or injury. Functional networks derived from fMRIs
typically superpose many neurophysiological responses elicited by stimuli.
Identifying these responses and separating associated functional networks into
their basic building blocks is essential to understanding the shared, and
unique aspects of neuronal responses across heterogeneous populations
performing different tasks. Ideally, these canonical networks must be grounded
in neurophysiology, identifying coherent modules of neural responses that are
interpretable by neuroscientists and other domain experts.

The method of choice for connectomic analysis is Independent Component
Analysis (ICA) [1, 2], which is used on individual fMRIs to spatially localize
regions of interest. Group-ICA [3, 4, 5] combines fMRIs across individuals to
model shared regions of interest. Other ML-based interpretable methods have
been proposed in the recent past [6, 7, 8, 9]. However, these methods are
limited in their ability to handle large datasets with diverse subjects (young
v/s old, healthy v/s diseased) performing a variety of cognitive tasks.
Moreover, most of these learned representations lack inherent interpretability
or transparency to yield insights into brain elemental networks associated
with different tasks. Large-scale efforts, such as the Human Connectome
Project [10], Cambridge Centre for Ageing and Neuroscience (CamCAN) [11], and
Alzheimer’s Disease Neuroimaging Initiative (ADNI) [12] have each collected
and curated neuroimages from cohorts of several hundred subjects. Each of
these datasets includes subjects of different ages, stages of neuroplasticity,
and degeneration. These datesets provide us with excellent opportunities for
developing methods for identifying canonical functional networks, their
compositions, and use in diffrent applications.

In this paper, we propose a novel framework for deconvolving networks derived
from fMRIs of subjects performing different tasks into a small set of
canonical networks that serve as building blocks that are: (i) shared across
large cohorts; (ii) can be composed into task-specific networks; and (iii) are
predictive of tasks and efficacy. We call these networks _canonical task
connectomes_. Our framework also characterizes the extent of expression of
these networks for each task, along with its neurophysiological basis.

Our approach first combines individual functional networks into a population-
level matrix $\mathbf{X}$. We then deconvolve this matrix into its factors
$\mathbf{W}$ and $\mathbf{H}$ such that each column $\mathbf{W}_{(\ast,i)}$
corresponds to a canonical task connectome, and the corresponding row
$\mathbf{H}_{(j,\ast)}$ characterizes the extent to which the canonical
network is expressed in every subject. However, since individual samples
(fMRIs) correspond to subjects performing different tasks, the latent
canonical representations must encode this important information. We
accomplish this by formulating and solving a suitable matrix factorization
problem.

We present our main experimental results on 1000 subjects from HCP at rest and
for six tasks (Language, Emotional Processing, Gambling, Motor, Relational
Processing, and Social Processing). Our results show that:

  * •

_Canonical task connectomes have high task-specificity_. We show that our
approach constructs networks that uniquely characterize different tasks and
are therefore excellent markers of tasks.

  * •

_Canonical task connectomes are generalizable across cohorts._ We show that
canonical representations obtained on a suitably constructed train set can
accurately predict tasks being performed by the test set.

  * •

_Canonical task connectomes identify common neural processes._ We show that
our approach finds canonical functional networks that are shared across tasks.
This enables novel interpretations of processes and responses associated with
different task stimuli.

  * •

_Canonical task connectomes have a strong physiological basis._ We show that
the canonical connectomes can be mapped to regions of the brain to identify
physiological underpinnings of tasks, that are in strong agreement with
literature in neurosciences.

The rest of the paper is organized as follows: in Section 2.1, we summarize
non-negative matrix factorization and its use in our context. In Section 2.2,
we provide details for our proposed framework. Then, we describe the HCP
dataset and the preprocessing pipeline in Section 2.3. This is followed by
comprehensive experimental validation in section 3, where we demonstrate the
interpretability and generalizability of our proposed approach. Finally, we
conclude with related work in Section 4 and summarize our contributions and
avenues for future work in Section 5.

##  2 Methods and Materials

We describe our formulation and solution to the problem of identifying
interpretable task-specific brain networks, called “connectomes” from
neuroimaging datasets of subjects performing a variety of cognitive tasks.
Connectomes are networks in which regions of the brain correspond to nodes and
correlated activity quantifies the strength of edges across corresponding
nodes (regions). We describe, in more detail, the process of construction of
connectomes in Section 2.3.

We hypothesize and validate that neuronal activity observed during a task is
composed of a small set of canonical patterns or motifs that recur across
subjects. Correspondingly, the overall observed connectome is a superposition
of these motifs that we call canonical task connectomes. The goal of our
formulation and methods is to demonstrate the existence and applications of
such canonical task connectomes.

We abstract the connectome into a $region\times region$ similarity matrix.
Applying non-negative matrix factorization to connectome data from a large
group of subjects, we factor the observed composite connectome into canonical
task connectomes. These factors, extracted by non-negative matrix
factorization (NMF), provide consistent and strong associations with specific
cognitive tasks. This provides strong evidence for our framework of canonical
task connectomes, while yielding physiologically interpretable results.

###  2.1 Non-negative Matrix Factorization (NMF)

Let $\mathbf{X}\in\mathbb{R}^{p\times n}_{\geq 0}$ denote the data matrix. NMF
approximately factorizes the data matrix $\mathbf{X}$ into two non-negative
matrices $\mathbf{W}$ and $\mathbf{H}$ by optimizing:

$\operatorname*{arg\,min}_{\mathbf{W},\mathbf{H}\geq
0}||\mathbf{X}-\mathbf{W}\mathbf{H}||_{F}^{2}$ (1)

Where, $\mathbf{W}\in\mathbb{R}^{p\times r}_{\geq 0}$ is the (non-negative)
“basis matrix” which is a low-rank, latent description of the columns of
$\mathbf{X}$, $\mathbf{H}\in\mathbb{R}^{r\times n}_{\geq 0}$ is the (non-
negative) “encoding matrix” matrix of coefficients that provides the weights
to each of the latent factors required to explain each data-point. Here, $r$
is the rank, representing the number of latent factors used to approximate the
original data matrix $\mathbf{X}$.

The non-negativity constraints in NMF lead to a parts-based representation of
the data unlike ICA, which allows positive and negative values. This enforces
sparsity and non-subtractive combinations:

$\mathbf{X}_{j}=\sum_{i=1}^{r}\mathbf{H}_{ji}\mathbf{W}_{i},\quad\forall\mathbf{H}_{ji}\geq
0$ (2)

where, $\mathbf{X}_{j}$ is the j-th column of matrix $\mathbf{X}$, and
$\mathbf{W}_{i}$ is the i-th column of matrix $\mathbf{W}$.

NMF decomposes the data into components that are additive and non-negative,
which is well-suited to fMRI data. Signals in fMRI data are usually additive
(i.e., the total signal is a sum of signals from different sources).
Furthermore, components identified by NMF can be directly mapped to specific
brain regions or networks, providing a clear and interpretable
characterization of brain activity.

###  2.2 Overview of proposed framework

In Fig. 1, we illustrate the general framework. We vectorize the connectomes
and stack them column-wise into a population-level data matrix
$\mathbf{X}\in\mathbb{R}^{\mathcal{O}(d^{2})\times n}$, $d$ denotes the number
of regions in each connectome, $n$ denotes the total number of data samples.

We decompose the matrix $\mathbf{X}$ into two matrices $\mathbf{W}$ and
$\mathbf{H}$ using NMF, as discussed in Section 2.1. The columns of the basis
matrix $\mathbf{W}$ represent canonical connectomes that can be linearly
combined to approximate the original functional connectomes in $\mathbf{X}$.
Columns of matrix $\mathbf{W}$ correspond to "canonical task connectomes"
since they capture fundamental networks across various subjects.

To validate the generalizability of these canonical representations, we divide
the cohort randomly into train and test sets. We first use training data to
compute the canonical task connectomes via NMF. These connectomes serve as an
interpretable basis to represent brain activity. For the test set, we find the
coefficients that best reconstruct each test sample using the pre-computed
connectome basis. We then train a model to map from coefficient patterns to
cognitive task labels using the training data. Finally, we apply this model
and use the resulting coefficients to predict the cognitive tasks performed by
subjects in the test set.

Our results show that canonical task connectomes strongly correlate with
anatomical and physiological processes associated with different cognitive
tasks. The ability to accurately predict tasks in new test data demonstrates
that the canonical connectomes generalize across subjects and capture robust
task-specific functional networks.

Figure 1: _Overview of proposed framework:_ (1) The training phase deconvolves
the data matrix of vectorized connectomes in the training set into a small
number of basis vectors; (2) The testing phase computes the coefficients of
the functional basis and predicts the task on new subjects.

###  2.3 Data

We validate our model and methods using data from 1000 subjects’ fMRIs from
the Human Connectome Project (HCP) Young Adult dataset [10]. For each subject,
we have separate fMRIs when they are at rest, and while they perform six
cognitive tasks (Language, Emotional Processing, Gambling, Motor, Relational
Processing, and Social Processing) [13]. We first use the Minimal Pre-
Processing Pipeline prescribed by the HCP consortium [14]. This process
includes spatial artifact/distortion removal, head motion correction,
registration, and normalization to standard space. For each input noisy fMRI,
the Minimal Preprocessing Pipeline outputs a clean and standardized
$voxel\times time$ time-series matrix. Then, we use the Atlas of Glasser et
al. [15] to aggregate this matrix into a $region\times time$ matrix. We note
that each _region_ consists of proximal _voxel_ s with shared anatomy. In all,
the Glasser Atlas demarcates 180 regions in each hemisphere (360 in total). We
then create the functional connectome (FC) matrix for each fMRI by computing
the Pearson Correlation between all pairs of regions. In all, we have 7000 FCs
($1000\ subjects\times 7\ tasks$). We vectorize the upper triangular matrix of
each FC and stack them side by side to create a population-level matrix of
size $64620\times 7000$.

##  3 Results

In this section, we show that our canonical task connectomes are conserved
across populations, highly specific to a small subset of tasks, and as a
consequence provide both an understanding of the neural response, as well as
the ability to predict tasks. We provide evidence of strong spatial
localization for these representative brain networks, which establishes
interpretability on the basis of neuro-anatomy. We also show that the regions
implicated in the tasks are supported by prior experimental studies, which
establishes physiological interpretations.

###  3.1 Canonical Task Connectomes are Generalizable across Cohorts

We show that canonical task connectomes are stable representations of
different tasks. To demonstrate this, we first compute the canonical
representations on a training set. We then predict the task performed in the
test set. Specifically, we create $\mathbf{X}_{train}$ and $\mathbf{X}_{test}$
by 80/20 random splits of the subjects. We decompose $\mathbf{X}_{train}$ to
find the canonical task connectomes $\tilde{\mathbf{W}}$ and the coefficient
matrix $\tilde{\mathbf{H}}$, and use $\tilde{\mathbf{H}}$ along with
corresponding task labels to train a classifier. Now, given a test subject (or
test set), we compute the least-squares solution $\hat{\mathbf{H}}$, the
coefficients corresponding to the test subjects, using
$\hat{\mathbf{H}}=\tilde{\mathbf{W}}^{\dagger}\mathbf{X}_{test}$, where
$\tilde{\mathbf{W}}^{\dagger}$ is the pseudo-inverse of the matrix of
canonical task connectomes from the test set. Finally, we predict the labels
of $\mathbf{X}_{test}$ using $\hat{\mathbf{H}}$ and the trained classifier.

Parameter study of rank: To determine the optimal rank, we decompose the data
matrix using NMF with rank ranging from 1 to 100. We then test the accuracy of
predicting the task labels from the factorized matrices. Figure 2 illustrates
the prediction accuracy with increasing rank. As shown, the accuracy increases
rapidly from rank=1 to 20, increasing from approximately 18% to 92%.
Subsequently, it exhibits a gradual rise, reaching approximately 98% once the
rank surpasses 50. For our main results in this paper, we select rank=20 as it
marks an inflection point and facilitates interpretation, without overfitting
to data.

Figure 2: _Parameter study: effect of rank on test accuracy._ We set rank from
1 to 100 and use NMF to deconvolve the train data matrix. We then test task
predictions based on $\hat{\mathbf{H}}$ using
$\hat{\mathbf{H}}=\tilde{\mathbf{W}}^{\dagger}\mathbf{X}_{test}$. There is
rapid increase in accuracy from rank 1 up to 20, after which the rate of
increase slows considerably. The accuracy plateaus around 98% at rank 50.

We demonstrate the effectiveness of our method for task prediction and feature
selection using three simple classifiers: K-nearest neighbor (KNN), support
vector machine (SVM), and a 3-layer multilayer perceptron (MLP). Our approach
achieves excellent performance on task prediction and is able to identify the
most relevant features for each task.

In Table 1, the first row summarizes the results for rank-20 approximations,
averaged over 10 runs. The factors computed by NMF yield high prediction test
accuracy by all three classifiers. In the following rows, we demonstrate the
strong feature selection ability of our method. We sort and eliminate the 90%,
95% and 99% smallest values in the 64620 feature dimensions of
$\mathbf{W}_{i}$, for $i\in\\{1,...,20\\}$.Then, we combine the remaining
dimensions of each $\mathbf{W}{i}$ to obtain a region-of-interest matrix with
only $\sim$52000, $\sim$37000, and $\sim$10500 features, reducing $\sim$19.5%,
$\sim$42.7% and $\sim$83.75% of the original data matrix. We use significantly
reduced set of features selected by our method for task prediction by the same
process above. Remarkably, results consistently outperform the original
feature set, yielding superior accuracy. We observe that a cut-off of 95% -
99% is ideal. To further demonstrate task-specific feature selection, we
perform binary classification on the matrix deconvolved only on the specific
task matrix $64620\times 1000$ rather than the entire $64620\times 7000$ data
matrix. We also sort and eliminate 90%, 95%, and 99% of the smallest values in
$\mathbf{W}_{i}$, for $i\in\\{1,...,20\\}$, and use the remaining non-zero
dimensions for binary task prediction, with the target task label = 1 and
other 6 tasks’ labels represented as 0. Table 2 shows the result of binary
classification using a 3-layer multilayer perceptron, yielding excellent
accuracy $\sim$98% with our feature selection.

Table 1: _Task prediction test accuracy using different classifiers._ | KNN | MLP | SVM
---|---|---|---
Original | 86.41 $\pm$ 1.05 | 92.31 $\pm$ 0.65 | 91.39 $\pm$ 1.18
Cut 90 | 86.41 $\pm$ 1.41 | 92.26 $\pm$ 0.97 | 91.99 $\pm$ 1.01
Cut 95 | 87.24 $\pm$ 1.59 | 93.16 $\pm$ 0.50 | 92.80 $\pm$ 0.94
Cut 99 | 87.77 $\pm$ 1.31 | 93.13 $\pm$ 0.81 | 92.39 $\pm$ 0.79
Table 2: _Binary classification test accuracy using MLP_ |  Rest |  Language |  Emotion |  Gambling |  Motor |  Relational |  Social
---|---|---|---|---|---|---|---
Original |  97.30$\pm$0.40 |  98.25$\pm$1.01 |  97.49$\pm$1.41 |  96.83$\pm$1.72 |  96.92$\pm$1.59 |  96.98$\pm$1.45 |  97.36$\pm$1.64
Cut 90 |  97.60$\pm$0.31 |  98.43$\pm$0.87 |  97.61$\pm$1.45 |  96.96$\pm$1.71 |  97.06$\pm$1.56 |  97.11$\pm$1.44 |  97.47$\pm$1.61
Cut 95 |  97.34$\pm$0.49 |  98.42$\pm$1.14 |  97.57$\pm$1.60 |  97.14$\pm$1.61 |  97.12$\pm$1.45 |  97.25$\pm$1.36 |  97.61$\pm$1.53
Cut 99 |  96.10$\pm$0.35 |  97.52$\pm$1.46 |  96.95$\pm$1.48 |  96.52$\pm$1.50 |  96.42$\pm$1.40 |  96.67$\pm$1.41 |  97.12$\pm$1.69

###  3.2 Canonical Task Connectomes are Building Blocks of Cognitive Tasks

We show that canonical task connectomes constitute fundamental building blocks
of human cognitive tasks. In the first set of results, we demonstrate that our
canonical connectomes encode information that constitutes the observed
composite connectome in a manner that is conserved across subjects. This is
non-trivial because of: a) inherent heterogeneity in basal brain activity
across individuals; b) individual-level differences in cognitive processes to
perform a task; c) diversity of task conditions; and d) noise in the imaging
modality. Using NMF, we deconvolve the population-level matrix $\mathbf{X}$.
As mentioned in Section 2.2, NMF outputs a) a small set of vectorized
canonical connectomes $\mathbf{W}$, and b) a linear coefficients matrix
$\mathbf{H}$. Each column of $\mathbf{H}_{(\ast,j)}$ represents the extent to
which the canonical connectomes contribute to the corresponding vectorized
connectome in $\mathbf{X}_{(\ast,j)}$. Additionally, we note that each row
$\mathbf{H}_{(i,\ast)}$ represents the linear coefficients for the $i-$th
canonical connectome, i.e. $\mathbf{W}_{(\ast,i)}$. In our experiments, we set
the rank to 20 to extract a compact set of informative task networks.

We z-score normalize the columns of $\mathbf{H}$ returned by NMF and retain
only those non-zero values higher than 90 percentile. The rows of
$\mathbf{H}$, grouped according to tasks are visualized in Fig. 3. It is
evident that the non-zeros of these significant coefficients are highly
selective of tasks. We use the 90th percentile as a cutoff to discard small
values and select the most significant coefficients. This allows us to
identify the canonical connectomes most strongly associated with each task (at
the 90th percentile significance level) which can be seen in Table 3.

Table 3: _Significant Connectomes for Each Cognitive Task_ Task | Significant Connectomes
---|---
Rest | W3, W8, W9, W11, W15, W18, W20
Language | W14, W17
Emotion | W7, W13, W20
Gambling | W6, W7, W13, W20
Motion | W1, W2, W4, W5, W11, W12, W18
Relational | W6, W13
Social | W4, W8, W10, W16, W19
Figure 3: _Coefficients matrix $\mathbf{H}$ of NMF._ We apply NMF to factor
the data matrix of rank 20. We fit columns of $\mathbf{H}$ to a normal
distribution and use the 90th percentile as a cutoff to discard small values.
Each plot shows the encoding of 1000 subjects performing one task, visualizing
$\mathbf{W}_{i}$ (corresponding to the canonical connectomes) that make up
that particular task.

We compute the similarity of fMRI responses for different tasks by
constructing a network in which nodes correspond to tasks and edges correspond
to similarities computed using our coefficient vectors. Specifically, if the
coefficient vectors for two tasks are similar, they are composed of the same
set of canonical connectomes in similar proportion. To demonstrate this, we
begin by averaging each of the seven cognitive tasks’ coefficient vectors
across 1000 subjects, obtaining a set of task-specific vectors from
$\tilde{\mathbf{H}}$. We then calculate the Pearson correlation coefficients
between these vectors and construct the task similarity network. Fig. 4 shows
the task similarity network along with the canonical connectomes associated
with each task. This network reveals high similarity between Emotion,
Relational, and Gambling tasks. Rest, Emotion and Motor are somewhat related,
while Social and Language tasks exhibit distinct characteristics from the
others, indicating distinct brain network engagements for these functions.

Figure 4: _Correlation network analysis._ We take the mean of each task’s
corresponding linear coefficients and compute the Pearson coefficient across
these averaged task vectors. We then construct the similarity network using
the Pearson coefficient and connect each task with its most significant
canonical connectomes.

###  3.3 Canonical Task Connectomes have a Strong Anatomical and Physiological
Basis

We show that: a) each canonical task connectome is spatially localized to
anatomically demarcated regions; and b) the regions enriched in each canonical
connectome are known to be implicated in the corresponding task. As before, we
deconvolve the population-level matrix $\mathbf{X}$ to compute $\mathbf{W}$
and $\mathbf{H}$. In this experiment, we use rank 20 approximation for
interpretation. From each column $\mathbf{W}_{i}$, we construct $region\times
region$ canonical task connectome, which is a correlation matrix and we sum
the rows to compute the active factor for each region and retain the top 2% of
them. We then use MRIcroGL [16] for visualization of the top active region for
each task.

Figure 5: _Canonical Task Connectomes have strong anatomical basis._ From each
column $\mathbf{W}_{i}$, we construct $region\times region$ canonical task
connectome. The correlation values for the 360 regions are summed as an active
factor, and the top 2% of highly active regions are retained. MRIcroGL is then
used to visualize the most active regions for each task.

In Fig 5, we visualize the task-specific connectomes by combining canonical
task connectomes that we find significant for each task in Section 3.2.

Our results indicate that anatomical regions indicative of rest correspond to
the dorsomedial prefrontal cortex, posterior cingulate cortex (in the limbic
node), the angular gyrus found in the posterior part of the inferior parietal
lobe, and substructures corresponding to the dorsal medial prefrontal cortex.
These regions are also implicated in the dorsal Default Mode Network (dDMN) –
a functional network known to be active during Rest [17]. In the language
processing task, our approach identifies regions in the inferior parietal
cortex/angular gyrus, which are well known to be active during language
processing tasks [18]. Further, regions in the left prefrontal cortex are
associated with word and sentence comprehension [19]. We also observed regions
in the auditory cortex and auditory association cortex which explains the
response to auditory stories. The regions implicated for emotion processing in
our setup include anterior cingulate and ventromedial prefrontal cortex, which
are both cortical regions known to be active when processing facial
expressions. Subcortical regions such as Amygdala, also commonly reported to
be active during emotional processing are not considered as we limit our study
to the cortex [20]. Since gambling tasks are cognitively complex, several
brain regions are co-activated. Our results demonstrate that subregions within
posterior and anterior cingulate, orbital frontal cortex and dorsolateral
prefrontal cortex are all implicated in gambling related tasks, which is in
line with observations of Li et al. [21]. The regions implicated in relational
processing are dorsolateral Prefrontal Cortex rostrolateral prefrontal cortex
and posterior parietal cortex, which are consistent with previous literature
[22]. The regions implicated in social processing are the medial prefrontal
cortex, which is located in the prefrontal cortex of the frontal lobe [23],
fusiform gyrus, and anterior cingulate cortex [24].

##  4 Related Work

Matrix Factorization. Independent Component Analysis (ICA) and its variants
are widely used in fMRI analysis. Spatial Independent Component Analysis (ICA)
[1, 25, 26, 27, 28] methods decompose fMRI data into a set of spatially
independent components. They identify patterns of activity across the brain
that are independent of one another. This information is used to identify
distinct networks of brain regions involved in various cognitive processes. In
a typical ICA model, the source signals are assumed to be statistically
independent and non-Gaussian, with an unknown linear mixing process. The model
assumes that every observed vector $x\in\mathbb{R}^{m}$ is generated by a
linear mixture of $n$ independent sources $x=\mathbf{A}s$, where
$s\in\mathbb{R}^{n}$ is an N-dimensional vector whose elements are the random
variables that refer to the independent sources and
$\mathbf{A}\in\mathbb{R}^{m\times n}$ is an unknown mixing matrix. ICA aims to
estimate an unmixing matrix $\mathbf{W}\in\mathbb{R}^{n\times m}$ such that
the recovered sources: $y=\mathbf{W}x=\mathbf{WA}s$ is a good representation
of the true sources $s$. Applying the typical ICA model to fMRI data, we have
data $\mathbf{X}=\mathbf{AS}$, where $\mathbf{X}\in\mathbb{R}^{N\times V}$
spans $N$ time points and $V$ voxels, and $\mathbf{S}$ contains spatially
independent source signals.

Group ICA is an extension of spatial ICA that allows the identification of
common patterns of activity across multiple subjects in a study. A popular
implementation of Group ICA is Multivariate Exploratory Linear Decomposition
into Independent Components (MELODIC) [29], which is part of the fMRI Standard
Library (FSL). Other approaches for multi-subject analysis using ICA have been
proposed [30, 31, 32, 33, 34]. The model in Calhoun et al. [30] defines Group
ICA as $\mathbf{X}_{i}=\mathbf{A}_{i}\mathbf{S}$, where
$\mathbf{X}_{i}\in\mathbb{R}^{N_{i}\times V}$ is the fMRI observation for
subject $i$ with $N_{i}$ time points and $V$ voxels. Group ICA captures a
group subspace with independent spatial maps and time courses. Then, these are
used to reconstruct subject-specific spatial maps $\mathbf{S}_{i}$ and time
courses $\mathbf{A}_{i}$. Group ICA has been widely used to study functional
connectivity differences between groups of healthy and clinical populations
[35, 36], as well as to identify brain networks associated with specific
cognitive processes across a group of individuals [37, 38]. We rely on NMF in
this study instead of more commonly used ICA and related methods, because NMF
requires that all components in the decomposition be strictly positive, which
along with sparsity of factors yields interpretability.

Deep learning methods. Deep learning models like MLPs and CNNs have been used
to extract high-level fMRI features for classification [39, 40]. More
commonly, classifiers use functional connectivity features between brain ROIs
to improve performance [41, 42]. Graph neural networks, with ROIs as nodes and
FC as edge weights, are gaining popularity for their alignment with the
brain’s structure [43, 44]. Recurrent and transformer models have also been
added to capture temporal dynamics in functional connectivity [45]. However,
with the increasing size of parameters, interpretability of these networks
remains an unresolved issue. Our framework explains observed (composite) brain
activity in terms of canonical task connectomes, which have biological basis –
a key contribution of our work.

Other methods. Subspace clustering methods are used in fMRI to partition data
into subspaces and assign each data point (e.g., voxel or region of interest)
to its corresponding subspace. This allows for the identification of different
brain activity patterns or functional connectivity profiles within data.
Several subspace clustering methods have been applied to fMRI data such as
spectral clustering [46, 47, 48], sparse subspace clustering [49, 50], low-
rank and sparse decomposition (LRSD) [51, 52, 53]. Subspace clustering reveals
distinct brain activity patterns, functional networks, or connectivity
profiles within the data. In our study, NMF is preferable to subspace
clustering for fMRI analysis due to its simplicity, interpretability, and
computational efficiency. The nonnegative parts-based decomposition from NMF
provides intuitive and meaningful canonical task connectomes that subspace
clustering methods cannot.

##  5 Conclusion

We presented a new problem and framework for fMRI analysis that deconvolves an
input set of neuroimages of subjects performing different cognitive tasks into
a compact set of task-specific elementary networks called “canonical task
connectomes”. Our approach formulates the problem as one of matrix
factorization, revealing that the resulting latent factors/networks can serve
as fundamental “building blocks” for various cognitive tasks. Experimental
results conducted on the Human Connectome Project dataset demonstrate how NMF
captures the inherent task-specific structure in suitably abstracted
neuroimages. Furthermore, we illustrate the utility of these canonical task
connectomes as effective biomarkers for predicting the performed task.
Additionally, we show anatomical and physiological underpinnings for the
networks identified by our framework. Our framework can be extended to more
complex applications, such as: a) understanding shared and unique functional
networks across different pathologies; and b) how task-specific networks can
get dysregulated due to the onset, and progression of diseases.

## References

  * [1] Scott Makeig, Anthony Bell, Tzyy-Ping Jung, and Terrence J Sejnowski.  Independent component analysis of electroencephalographic data.  Advances in neural information processing systems, 8, 1995.
  * [2] Aapo Hyvärinen and Erkki Oja.  Independent component analysis: algorithms and applications.  Neural networks, 13(4-5):411–430, 2000.
  * [3] Fabrizio Esposito, Tommaso Scarabino, Aapo Hyvarinen, Johan Himberg, Elia Formisano, Silvia Comani, Gioacchino Tedeschi, Rainer Goebel, Erich Seifritz, and Francesco Di Salle.  Independent component analysis of fmri group studies by self-organizing clustering.  Neuroimage, 25(1):193–205, 2005.
  * [4] Yuhui Du and Yong Fan.  Group information guided ica for fmri data analysis.  Neuroimage, 69:157–197, 2013.
  * [5] Qiu-Hua Lin, Jingyu Liu, Yong-Rui Zheng, Hualou Liang, and Vince D Calhoun.  Semiblind spatial ica of fmri using spatial constraints.  Human brain mapping, 31(7):1076–1088, 2010.
  * [6] Logan Grosenick, Stephanie Greer, and Brian Knutson.  Interpretable classifiers for fmri improve prediction of purchases.  IEEE transactions on neural systems and rehabilitation engineering, 16(6):539–548, 2008.
  * [7] Shailee Jain, Vy Vo, Shivangi Mahto, Amanda LeBel, Javier S Turek, and Alexander Huth.  Interpretable multi-timescale models for predicting fmri responses to continuous natural speech.  Advances in Neural Information Processing Systems, 33:13738–13749, 2020.
  * [8] Xiaoxiao Li, Yuan Zhou, Nicha Dvornek, Muhan Zhang, Siyuan Gao, Juntang Zhuang, Dustin Scheinost, Lawrence H Staib, Pamela Ventola, and James S Duncan.  Braingnn: Interpretable brain graph neural network for fmri analysis.  Medical Image Analysis, 74:102233, 2021.
  * [9] Xuan Kan, Hejie Cui, Joshua Lukemire, Ying Guo, and Carl Yang.  Fbnetgen: Task-aware gnn-based fmri analysis via functional brain network generation.  In International Conference on Medical Imaging with Deep Learning, pages 618–637. PMLR, 2022.
  * [10] David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, Wu-Minn HCP Consortium, et al.  The wu-minn human connectome project: an overview.  Neuroimage, 80:62–79, 2013.
  * [11] Jason R Taylor, Nitin Williams, Rhodri Cusack, Tibor Auer, Meredith A Shafto, Marie Dixon, Lorraine K Tyler, Richard N Henson, et al.  The cambridge centre for ageing and neuroscience (cam-can) data repository: Structural and functional mri, meg, and cognitive data from a cross-sectional adult lifespan sample.  neuroimage, 144:262–269, 2017.
  * [12] Clifford R Jack Jr, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski, Paula J Britson, Jennifer L. Whitwell, Chadwick Ward, et al.  The alzheimer’s disease neuroimaging initiative (adni): Mri methods.  Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine, 27(4):685–691, 2008.
  * [13] Deanna M Barch, Gregory C Burgess, Michael P Harms, Steven E Petersen, Bradley L Schlaggar, Maurizio Corbetta, Matthew F Glasser, Sandra Curtiss, Sachin Dixit, Cindy Feldt, et al.  Function in the human connectome: task-fmri and individual differences in behavior.  Neuroimage, 80:169–189, 2013.
  * [14] Matthew F Glasser, Stamatios N Sotiropoulos, J Anthony Wilson, Timothy S Coalson, Bruce Fischl, Jesper L Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R Polimeni, et al.  The minimal preprocessing pipelines for the human connectome project.  Neuroimage, 80:105–124, 2013.
  * [15] Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al.  A multi-modal parcellation of human cerebral cortex.  Nature, 536(7615):171–178, 2016.
  * [16] Chris Rorden and Matthew Brett.  Stereotaxic display of brain lesions.  Behavioural neurology, 12(4):191–200, 2000.
  * [17] Bharat Biswal, F Zerrin Yetkin, Victor M Haughton, and James S Hyde.  Functional connectivity in the motor cortex of resting human brain using echo-planar mri.  Magnetic resonance in medicine, 34(4):537–541, 1995.
  * [18] Mohamed L Seghier.  The angular gyrus: multiple functions and multiple subdivisions.  The Neuroscientist, 19(1):43–61, 2013.
  * [19] John DE Gabrieli, Russell A Poldrack, and John E Desmond.  The role of left prefrontal cortex in language and memory.  Proceedings of the national Academy of Sciences, 95(3):906–913, 1998.
  * [20] Goran Šimić, Mladenka Tkalčić, Vana Vukić, Damir Mulc, Ena Španić, Marina Šagud, Francisco E Olucha-Bordonau, Mario Vukšić, and Patrick R. Hof.  Understanding emotions: Origins and roles of the amygdala.  Biomolecules, 11(6):823, 2021.
  * [21] Xiangrui Li, Zhong-Lin Lu, Arnaud D’Argembeau, Marie Ng, and Antoine Bechara.  The iowa gambling task in fmri images.  Human brain mapping, 31(3):410–423, 2010.
  * [22] Keith J Holyoak and Martin M Monti.  Relational integration in the human brain: A review and synthesis.  Journal of cognitive neuroscience, 33(3):341–356, 2021.
  * [23] Chris D Frith.  The social brain?  Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1480):671–678, 2007.
  * [24] Ivy F Tso, Saige Rutherford, Yu Fang, Mike Angstadt, and Stephan F Taylor.  The “social brain” is highly sensitive to the mere presence of social information: An automated meta-analysis and an independent study.  PloS one, 13(5):e0196503, 2018.
  * [25] Wei Zhang, Jinglei Lv, Xiang Li, Dajiang Zhu, Xi Jiang, Shu Zhang, Yu Zhao, Lei Guo, Jieping Ye, Dewen Hu, and Tianming Liu.  Experimental comparisons of sparse dictionary learning and independent component analysis for brain network inference from fmri data.  IEEE Transactions on Biomedical Engineering, 66:289–299, 2019.
  * [26] K. A. Smitha, K. M. Arun, P.G. Rajesh, Bejoy Thomas, Ashalatha Radhakrishnan, Prabhakaran Sankara Sarma, and C Kesavadas.  Resting fmri as an alternative for task-based fmri for language lateralization in temporal lobe epilepsy patients: a study using independent component analysis.  Neuroradiology, 61:803–810, 2019.
  * [27] Vince D. Calhoun and T. Adalı.  Multisubject independent component analysis of fmri: A decade of intrinsic networks, default mode, and neurodiagnostic discovery.  IEEE Reviews in Biomedical Engineering, 5:60–73, 2012.
  * [28] Vince D. Calhoun and T. Adalı.  Unmixing fmri with independent component analysis.  IEEE Engineering in Medicine and Biology Magazine, 25:79–90, 2006.
  * [29] Christian F Beckmann and Stephen M Smith.  Probabilistic independent component analysis for functional magnetic resonance imaging.  IEEE transactions on medical imaging, 23(2):137–152, 2004.
  * [30] Vince D. Calhoun, T. Adalı, Godfrey D. Pearlson, and James J. Pekar.  A method for making group inferences from functional mri data using independent component analysis.  Human Brain Mapping, 14, 2001.
  * [31] Fabrizio Esposito, Tommaso Scarabino, Aapo Hyvärinen, Johan Himberg, Elia Formisano, Silvia Comani, Gioacchino Tedeschi, Rainer Goebel, Erich Seifritz, and Francesco Di Salle.  Independent component analysis of fmri group studies by self-organizing clustering.  NeuroImage, 25:193–205, 2005.
  * [32] Ying Guo and Giuseppe Pagnoni.  A unified framework for group independent component analysis for multi-subject fmri data.  NeuroImage, 42:1078–1093, 2008.
  * [33] Vincent J. Schmithorst and Scott K Holland.  Comparison of three methods for generating group statistical inferences from independent component analysis of functional magnetic resonance imaging data.  Journal of Magnetic Resonance Imaging, 19, 2004.
  * [34] Christian F. Beckmann and Stephen M. Smith.  Tensorial extensions of independent component analysis for multisubject fmri analysis.  NeuroImage, 25:294–311, 2005.
  * [35] Xuan Vinh To, Viktor Vegh, and Fatima Ali Nasrallah.  Towards data-driven group inferences of resting-state fmri data in rodents: Comparison of group ica, gig-ica, and iva-gl.  Journal of Neuroscience Methods, 366, 2021.
  * [36] Yuhui Du and Yong Fan.  Group information guided ica for fmri data analysis.  NeuroImage, 69:157–197, 2013.
  * [37] Ahmed Abou Elseoud.  Exploring functional brain networks using independent component analysis : functional brain networks connectivity.  2013\.
  * [38] JeYoung Jung and Matthew. A. Lambon Ralph.  Distinct but cooperating brain networks supporting semantic cognition.  Cerebral Cortex (New York, NY), 33:2021 – 2036, 2021.
  * [39] Heung-Il Suk, Chong-Yaw Wee, Seong-Whan Lee, and Dinggang Shen.  State-space model with deep learning for functional dynamics estimation in resting-state fmri.  NeuroImage, 129:292–307, 2016.
  * [40] Heng Huang, Xintao Hu, Yu Zhao, Milad Makkie, Qinglin Dong, Shijie Zhao, Lei Guo, and Tianming Liu.  Modeling task fmri data via deep convolutional autoencoder.  IEEE transactions on medical imaging, 37(7):1551–1561, 2017.
  * [41] Regina J Meszlényi, Krisztian Buza, and Zoltán Vidnyánszky.  Resting state fmri functional connectivity-based classification using a convolutional neural network architecture.  Frontiers in neuroinformatics, 11:61, 2017.
  * [42] Xiaodan Xing, Qingfeng Li, Hao Wei, Minqing Zhang, Yiqiang Zhan, Xiang Sean Zhou, Zhong Xue, and Feng Shi.  Dynamic spectral graph convolution networks with assistant task training for early mci diagnosis.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 639–646. Springer, 2019.
  * [43] Xiaoxiao Li, Yuan Zhou, Nicha Dvornek, Muhan Zhang, Siyuan Gao, Juntang Zhuang, Dustin Scheinost, Lawrence H Staib, Pamela Ventola, and James S Duncan.  Braingnn: Interpretable brain graph neural network for fmri analysis.  Medical Image Analysis, 74:102233, 2021.
  * [44] Xiaoxiao Li, Nicha C Dvornek, Yuan Zhou, Juntang Zhuang, Pamela Ventola, and James S Duncan.  Graph neural network for interpreting task-fmri biomarkers.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part V 22, pages 485–493. Springer, 2019.
  * [45] Byung-Hoon Kim, Jong Chul Ye, and Jae-Jin Kim.  Learning dynamic graph representation of brain connectome with spatio-temporal attention.  Advances in Neural Information Processing Systems, 34:4314–4327, 2021.
  * [46] Sukrit Gupta and Jagath Rajapakse.  Iterative consensus spectral clustering improves detection of subject and group level brain functional modules.  Scientific Reports, 10, 2020.
  * [47] Richard Cameron Craddock, George Andrew James, Paul E. Holtzheimer, Xiaoping Hu, and Helen S. Mayberg.  A whole brain fmri atlas generated via spatially constrained spectral clustering.  Human Brain Mapping, 33, 2012.
  * [48] Esraa Al-sharoa, Mahmood A. Al-khassaweneh, and Selin Aviyente.  Temporal block spectral clustering for multi-layer temporal functional connectivity networks.  2018 IEEE Statistical Signal Processing Workshop (SSP), pages 503–507, 2018.
  * [49] Xiuchao Sui, Shaohua Li, and Jagath Rajapakse.  Locality regularized sparse subspace clustering with application to cortex parcellation on resting fmri.  2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pages 1286–1290, 2016.
  * [50] Seung-Jun Kim and Krishna Dontaraju.  Joint fmri analysis and subject clustering using sparse dictionary learning.  In Optical Engineering + Applications, 2017.
  * [51] Wei Tu, Fangfang Fu, Linglong Kong, Bei Jiang, Dana Cobzas, and Chao Huang.  Low-rank plus sparse decomposition of fmri data with application to alzheimer’s disease.  Frontiers in Neuroscience, 16, 2022.
  * [52] Eneko Uruñuela, Stefano Moia, and César Caballero-Gaudes.  A low rank and sparse paradigm free mapping algorithm for deconvolution of fmri data.  2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1726–1729, 2021.
  * [53] Vimal Singh, Ahmed H. Tewfik, and David B. Ress.  Under-sampled functional mri using low-rank plus sparse matrix decomposition.  2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 897–901, 2015.


# H-ensemble: an Information Theoretic Approach to
Reliable Few-Shot Multi-Source-Free Transfer

Yanru Wu 1, Jianning Wang 2, Weida Wang 1, Yang Li 1111Corresponding author.

###### Abstract

Multi-source transfer learning is an effective solution to data scarcity by
utilizing multiple source tasks for the learning of the target task. However,
access to source data and model details is limited in the era of commercial
models, giving rise to the setting of multi-source-free (MSF) transfer
learning that aims to leverage source domain knowledge without such access. As
a newly defined problem paradigm, MSF transfer learning remains largely
underexplored and not clearly formulated. In this work, we adopt an
information theoretic perspective on it and propose a framework named
H-ensemble, which dynamically learns the optimal linear combination, or
ensemble, of source models for the target task, using a generalization of
maximal correlation regression. The ensemble weights are optimized by
maximizing an information theoretic metric for transferability. Compared to
previous works, H-ensemble is characterized by: 1) its adaptability to a novel
and realistic MSF setting for few-shot target tasks, 2) theoretical
reliability, 3) a lightweight structure easy to interpret and adapt. Our
method is empirically validated by ablation studies, along with extensive
comparative analysis with other task ensemble and transfer learning methods.
We show that the H-ensemble can successfully learn the optimal task ensemble,
as well as outperform prior arts.

## Introduction

The scarcity of annotated data is a principal challenge to machine learning
algorithms in a variety of real-world scenarios, where transfer learning has
emerged as an effective remedy, harnessing information and insights from well-
established source tasks to augment data-deficient domains (see Perkins,
Salomon et al. 1992; Weiss, Khoshgoftaar, and Wang 2016; Zhuang et al. 2020).
Historically, transfer learning has centered on a singular source-task and
target-task relationship. However, the spotlight is gradually shifting towards
multi-source transfer learning, capitalizing on multiple source tasks to
facilitate the training of the target task (see Sun, Shi, and Wu 2015). A
variety of works have contributed to both the theoretical foundation and
practical algorithms on this topic. Yet, the trend of decentralization and
growing privacy concerns have impeded access to source data, particularly when
compared to the models they’ve nurtured (Feng et al. 2021). This evolving
landscape underscores the need for the novel paradigm of multi-source-free
(MSF) transfer learning (Fang et al. 2022).

MSF transfer learning, sometimes referred to as MSF domain adaptation (Han et
al. 2023), entails transferring knowledge from multiple sources to a target
task without accessing source data, as depicted in Fig. 1. Despite not being
clearly defined in most previous papers and still largely underexplored, MSF
transfer has been explored by several works these years. Preliminary efforts
centered around simple strategies like averaging source models or reducing the
problem to single source-free transfers, by source selection using empirical
validations or transferability metrics (e.g. Yang et al. 2023). Contemporary
research has pivoted towards sophisticated model ensembles, assigning weights
to sources based on perceived importance. For instance, approaches range from
correlating source relevance to the target (Lee, Sattigeri, and Wornell 2019),
to conditional entropy minimization (Ahmed et al. 2021), and even leveraging
source-similarity and class-relationship perception modules for weight
determination (Dong et al. 2021; Han et al. 2023). Beyond these linear
ensemble techniques, some researchers also introduced auxiliary models to
endow cross-domain capabilities (Li et al. 2022).

Figure 1: Problem Setting of Multi-Source-Free Transfer Learning. The transfer
to the target task is based on the source models (whose access in our
redefined MSF setting is also restricted) trained on the inaccessible source
data.

### Motivation

While these methods have their merits, they still present three distinct
limitations. Firstly, they mainly evaluate the source importance as an
independent characteristic of each source instead of viewing them as a whole,
neglecting the likely interaction effects when combining. Secondly, these
works focus on the unsupervised MSF transfer, while a more common setting of
data scarcity in practice is few-shot learning, where we can only access a few
of the target data but with labels (Wang et al. 2020). Thirdly, most of them
hinge on accessing intermediate layer outputs or necessitate tuning of source
models, which tend to become inaccessible in the era of proprietary large
models. Also, despite their effectiveness, these transfer algorithms have
quite complicated methodologies, making them hard to interpret or adapt.

As a solution to these issues, we focus on the few-shot MSF transfer
challenge, proposing a novel transfer framework tailored to this setting.
Given the scarcity of target samples and the inaccessibility of source
details, we base our methodology on the optimization of features instead of
the analysis of input data as other works did. Specifically, following the
idea of linear source ensemble, we define the target feature as a weighted
combination of transferred features and introduce an information theoretic
metric, H-score, to gauge the quality of this synthesis, which can be
alternatively interpreted as the transferability of the whole source model
ensemble. The optimal source weights will then be simply determined by
maximizing this metric, leveraging the robustness of H-score under few-shot
settings as well as its computational efficiency and theoretical reliability
(Huang et al. 2019b). Our derived target generalizes the maximal correlation
regression (MCR) by replacing its feature by an ensemble of source feature
extractors and modifying the model accordingly. A detailed explanation of the
underlying theoretic basis and intuition of H-ensemble framework will be
presented in the Methodology section, showing that: 1) under the maximal
correlation framework, the weights maximizing our adapted H-score are
theoretically optimal for the target task; 2) the resulting optimal question
is of a benign form and can be reliably solved by gradient descend based
algorithms.

Setting | Multi-Source | No Source Data | No Source Model Details | Target
---|---|---|---|---
Few-Shot transfer (Wang et al. 2020) | ✗ | ✗ | ✗ | Few-shot
Unsupervised DA (Hoffman et al. 2018) | ✗ | ✗ | ✗ | Unsupervised
Multi-Source DA (Peng et al. 2019) | ✓ | ✗ | ✗ | Unsupervised
Multi-Source-Free DA (Ahmed et al. 2021) | ✓ | ✓ | ✗ | Unsupervised
Few-Shot Multi-Source-Free Transfer (Ours) | ✓ | ✓ | ✓ | Few-shot
Table 1: Comparison across Relative Problem Settings, involving the aspects of
whether there are multi-sources, no source data, no source model details, and
different target learning approaches. The abbreviation ‘DA’ represents domain
adaptation, ‘✓’ represents obtaining the corresponding aspects, while ‘✗’ the
opposite. Among all settings, ours is relatively restrictive.

### Contributions

In this paper, we propose an information theoretic MSF transfer framework
named H-ensemble, where the source models are assembled using a generalization
of MCR, and the source weights are obtained by maximizing an H-score based
metric. Our contributions mainly lie in three aspects:

  * a)

We formulate the few-shot MSF transfer problem mathematically using a source-
free setting that is more common in practical scenarios.

  * b)

An efficient and theoretically reliable framework derived from maximal
correlation regression is introduced for MSF transfer. The framework is
designed to be lightweight and easy to interpret and adapt, possible to serve
as a plug-in module.

  * c)

We present both theoretical guarantee and experimental support to the
effectiveness of our methodology.

## Related Works

### Maximal Correlation Regression and H-Score

Maximal correlation regression (MCR) network (Xu and Huang 2020) refers to a
specially designed network trained based on the optimization of feature
quality by maximizing H-score. Here, H-score is a metric determining whether
certain features are informative for the task (more formally introduced by Bao
et al. 2019)222The H-score introduced by Bao et al. is actually the one-sided
form of the score in MCR (both firstly defined by Huang et al. in 2019). They
are also differed in their definition of variables and optimal objects.. Its
derivation is rooted in the maximal correlation interpretation of deep neural
networks (Huang et al. 2019a), where given the extracted features, the formula
of H-score is actually the optimal network performance under an information
theoretic measurement. Previous work also extended H-score to a
transferability metric and verified its effectiveness with extensive
experiments (Bao et al. 2019; Ibrahim, Ponomareva, and Mazumder 2022),
suggesting the potential of H-score in transfer learning.

Notably, as the basis of both MCR network and H-score, maximal correlation
analysis originates from the works of Hirschfeld, Gebelein and Renyi
(Hirschfeld 1935; Gebelein 1941; Rényi 1959) and has been followed and further
explored by a broad spectrum of successive work. A detailed overview of
maximal correlation analysis can be found given by Huang et al. (2019a).

### Multi-Source Transfer Learning

Multi-source transfer learning, a well-established concept, encompasses
approaches like naive multi-source transfer by feature extractor concatenation
(Christodoulidis et al. 2016) and theoretical explorations (Crammer, Kearns,
and Wortman 2008; Mansour, Mohri, and Rostamizadeh 2008; Ben-David et al.
2010; Tong et al. 2021; Chen and Marchand 2023). Contemporary algorithms can
be classified into three main categories: re-weighting approaches that
estimate and assign source importance (Sun et al. 2011; Lee, Sattigeri, and
Wornell 2019; Shui et al. 2021); source selection methods using
transferability metrics to identify optimal combinations (Agostinelli et al.
2022); and boosting-based solutions that apply ensemble learning techniques to
transfer learning (Huang, Wang, and Qin 2012; Xu and Sun 2012; Fang et al.
2019). Further insights on this topic are available in related surveys, such
as (Sun, Shi, and Wu 2015).

## Problem Definition

### Setting Comparison to Previous Works

Based on the change of settings mentioned in the Introduction section, there
are mainly three terms to emphasize in our setting: multi-source, source-free
and few-shot. Specially, to distinguish from previous notions, source-free is
categorized into source-data-free and source-model-free. We compare few-shot
MSF transfer with previous settings in these four aspects and summarize the
main divergence in Table 1.

Notation | Meaning
---|---
$S_{j}$; $T$ | Source/Target Task
$d_{S_{j}}$; $d_{T}$ | Source/Target Class Number
$N_{S_{j}}$; $N_{T}$ | Source/Target Data Size
$D_{S_{j}}=\\{(x_{S_{j}}^{i},y_{S_{j}}^{i})\\}_{i=1}^{N_{S_{j}}}$ | Source Data
$D_{T}=\\{(x_{T}^{i},y_{T}^{i})\\}_{i=1}^{N_{T}}$ | Target Data
Table 2: Notation. We omit $\\{\cdot\\}_{j=1}^{M}$ for simplification.

### Mathematical Formulation

Consider the transfer learning setting with $M$ source tasks
$\\{S_{j}\\}_{j=1}^{M}$ and a target task $T$, we list the main notations in
Table 2. For simplification, we assume that the sources and target tasks share
the same input space $X$ and all tasks are classification problems.333This
assumption is without loss of generality, for in most cases, model inputs can
be mapped to any size by preprocessing (e.g. image resizing, pooling, certain
embedding layers, etc.), and classification is one of the most representative
tasks in machine learning.

Notably, in our setting we have no access to source data $D_{S_{j}}$ and only
partial access to target data $D_{T}$, so here $N_{S_{j}}=0$, $N_{T}=k\times
d_{T}$ for a $k$-shot MSF transfer.444$k$ samples for each of the $d_{T}$
classes For the source model consisting of a backbone feature extractor
$\boldsymbol{f}_{S_{j}}:X\to\mathbb{R}^{d_{f}}$ and a classification head
$\boldsymbol{g}_{S_{j}}:\mathbb{R}^{d_{f}}\to\mathbb{R}^{d_{S_{j}}}$ ($d_{f}$
denotes the feature dimension)555$d_{f}$ is set to the same across all tasks
considering that we could add a fully connected layer to derive any output
dimension., where the source-free restriction leaves the detailed model
structure unknown.

In computation, we estimate the distribution $P^{T}_{X}$, $P^{T}_{Y}$ of
target data $x_{T}$, $y_{T}$ with empirical distribution $\hat{P}^{T}_{X}$,
$\hat{P}^{T}_{Y}$:

$\hat{P}_{X}^{T}(x)=\frac{1}{N_{T}}\sum_{i=1}^{N_{T}}\mathbb{I}\\{x_{T}^{i}=x\\};\hat{P}_{Y}^{T}(y)=\frac{1}{N_{T}}\sum_{i=1}^{N_{T}}\mathbb{I}\\{y_{T}^{i}=y\\},$
(1)

where $\mathbb{I}\\{\cdot\\}$ is the indicator function (Feller 1991).
Similarly, the conditional distribution of $x_{T}$ given $y_{T}$ $P^{T}_{X|Y}$
is estimated by its empirical distribution $\hat{P}^{T}_{X|Y}$:

$\hat{P}^{T}_{X|Y}(x|y)=\frac{\sum_{i=1}^{N_{T}}\mathbb{I}\\{x_{T}^{i}=x,y_{T}^{i}=y\\}}{\sum_{i=1}^{N_{T}}\mathbb{I}\\{y_{T}^{i}=y\\}}.$
(2)

When estimating the underlying distributions with empirical distributions of
samples, Eqns. (1) and (2) may be poorly defined and intractable for most
target task types due to the high dimension of input $x$. However, this
problem can be avoided, as in our framework we only need to compute the
expectation of features over these distributions.

The goal of few-shot MSF problem is that, by utilizing the partially
accessible data and black-box source models only, find the optimal parameters
$\boldsymbol{\theta}_{T}$ that minimizes certain divergence $\mathcal{L}$
between the real conditional distribution $P^{T}_{Y|X}$ and the approximated
$P^{(\boldsymbol{\theta}_{T})}_{Y|X}$, i.e.,

$\boldsymbol{\theta}^{*}_{T}=arg\min_{\boldsymbol{\theta}_{T}}\mathcal{L}(P^{(\boldsymbol{\theta}_{T})}_{Y|X},P^{T}_{Y|X}).$
(3)

Figure 2: Framework of H-Ensemble. The framework consists of three modules.
The target data firstly flow into Target Feature Extractor. Then the Weight
Optimizer will utilize the outputs of source feature extractors and target
label to derive the optimal source weight $\boldsymbol{\alpha}$, which makes
the parameter in deriving target feature. Finally the Target Classifier will
be trained and used together with the extractor for test according to a
generalization of maximal correlation regression (MCR).

## Methodology

Following the analysis in the Motivation section, we model the target feature
extractor $\boldsymbol{f}_{T}$ by a weighted combination of source feature
extractors $\boldsymbol{f}_{S_{j}}$:

$\displaystyle\boldsymbol{f}_{T}=\sum_{j=1}^{M}\alpha_{j}\cdot\boldsymbol{f}_{s_{j}},\
\sum_{j=1}^{M}\alpha_{j}=1.$ (4)

We then need to efficiently evaluate target features to guide the optimization
of weights $\boldsymbol{\alpha}$, where the maximal correlation analysis (MCA)
is introduced. In MCA, a neural network is viewed as a feature extractor
$\boldsymbol{f}$ and a classifier $\boldsymbol{g}$, and the informativeness of
features extracted by $\boldsymbol{f}$ can be estimated by H-score.

Leveraging the robustness of H-score under few-shot settings as well as its
computational efficiency and theoretical reliability, we use H-score as the
feature quality measurement and set up the transfer framework under the same
theory scheme as a generalization of MCR. Following previous works (Xu and
Huang 2020), the divergence $\mathcal{L}$ in Eqn. (3) is defined as
variational chi-squared divergence:

$\mathcal{L}(\boldsymbol{\theta})=\sum_{x}P_{X}(x)\sum_{y}\frac{[P^{(\boldsymbol{\theta})}_{Y|X}(y|x)-P_{Y|X}(y|x)]^{2}}{P_{Y}(y)}.$
(5)

We illustrate our MSF transfer framework H-ensemble in Fig. 2 and introduce
each module in the sections below, showing how the divergence $\mathcal{L}$ is
minimized.

### Multi-Source-Free Transfer

Overall, the MSF transfer framework follows the generalized MCR(Xu and Huang
2020) network and has two modules, feature extractor
$\boldsymbol{f}_{T}=\sum_{j=1}^{M}\alpha_{j}\cdot\boldsymbol{f}_{s_{j}}$ and
classifier $\boldsymbol{g}_{T}$ derived using MCR training mode as below. The
approximated conditional distribution is defined as

$P^{(\boldsymbol{\theta_{T}})}_{Y|X}(y|x)=P^{T}_{Y}(y)(1+\boldsymbol{f}_{T}^{\rm
T}(x)\boldsymbol{g}_{T}(y)).$ (6)

Specifically, given input data $x\sim P^{T}_{X}$, the features extracted by
source models are $\boldsymbol{f}_{s_{j}}(x)$. To simplify the deduction, we
assume that $\mathbb{E}_{P^{T}_{X}}[\boldsymbol{f}_{s_{j}}(x)]=0$ and
$\mathbb{E}_{P^{T}_{X}}[\boldsymbol{f}_{s_{j}}(x)\cdot\boldsymbol{f}_{s_{j}}(x)^{\rm
T}]=\mathbf{I}$ for all $j$, as a normalization layer could be efficiently
added to extractors. Subsequently, $\boldsymbol{f}_{T}$ is also a normalized
feature with $\mathbb{E}_{P^{T}_{X}}[\boldsymbol{f}_{T}(x)]=0$ and
$\mathbb{E}_{P^{T}_{X}}[\boldsymbol{f}_{T}(x)\cdot\boldsymbol{f}_{T}(x)^{\rm
T}]=\mathbf{I}$. After deriving the target feature extractor
$\boldsymbol{f}_{T}$, the optimal target classifier $\boldsymbol{g}_{T}$
minimizing $\mathcal{L}(\boldsymbol{\theta}_{T})$ can be directly calculated
by generalizing MCR to a multiple extractor combination scenario.

######  Theorem 1.

Given feature extractors of function form $\boldsymbol{f}_{i}(x)$,
$i\in\\{1,\dots,n\\}$ and the linearly combined feature extractor
$\boldsymbol{f}(x)=\sum_{i=1}^{n}\alpha_{i}\cdot\boldsymbol{f}_{i}(x)$, where
$\mathbb{E}_{P_{X}}[\boldsymbol{f}_{i}(x)]=0$,
$\mathbb{E}_{P_{X}}[\boldsymbol{f}_{i}(x)\cdot\boldsymbol{f}_{i}(x)^{\rm
T}]=\mathbf{I},\forall i$ and $\sum_{i=1}^{n}\alpha_{i}=1$. The optimal
classifier $\boldsymbol{g}^{*}(y)$ will be of the form:

$\displaystyle\boldsymbol{g}^{*}(y)=\sum_{i=1}^{n}$
$\displaystyle\alpha_{i}\cdot\mathbb{E}_{P_{X|Y}}[\boldsymbol{f}_{i}(X)|Y=y],$
(7)

where the proof is provided in the Appendix.I.

By using Eqn. (7), we can skip training and directly set the target classifier
$\boldsymbol{g}_{T}(y)$ to the theoretical optimal
$\boldsymbol{g}_{T}^{*}(y)=\sum_{i=1}^{n}\alpha_{i}\cdot\mathbb{E}_{P_{X|Y}}[\boldsymbol{f}_{i}(X)|Y=y]$,
thereby achieving high efficiency in transfer.

Notably, the form of $\boldsymbol{g}_{T}(y)$ does not conform to the usual
classifiers that take in extracted features and output the class probabilities
(as in the source classifiers
$\boldsymbol{g}_{S_{j}}:\mathbb{R}^{d_{f}}\to\mathbb{R}^{d_{S_{j}}}$). On the
contrary, the target classifier $\boldsymbol{g}_{T}(y)$ acts as a label
encoder, which maps labels $y$ to a newly defined y-feature, i.e.
$\boldsymbol{g}_{T}:\mathbb{R}^{d_{T}}\to\mathbb{R}^{d_{f}}$. In testing mode,
the classification result is given by maximizing the correlation between the
features extracted, i.e. $\hat{y}(x)=arg\max_{y}\boldsymbol{f}^{\rm
T}_{T}(x)\boldsymbol{g}_{T}(y)$. More details of this network architecture are
discussed in previous works (Huang et al. 2019a).

Input: Target data $D_{T}=\\{(x_{T}^{i},y_{T}^{i})\\}_{i=1}^{N_{T}}$, source
feature extractors $\\{\boldsymbol{f}_{S_{j}}\\}_{j=1}^{M}$

Learning rate $\lambda$ Output: Target classifier $\boldsymbol{g}_{T}$, source
weight $\boldsymbol{\alpha}$

Randomly initialize
$\boldsymbol{\alpha}=\\{\alpha_{1},\alpha_{2},\dots,\alpha_{M}\\}$, $\ \
\sum_{j=1}^{M}\alpha_{j}=1$ ;

for  _$y\leftarrow 1$ to $d_{T}$_ do // Compute empirical conditional
expectation

$n_{y}\leftarrow\sum_{i=1}^{N_{T}}\mathbb{I}\\{y=y^{i}_{T}\\}$ ;

$\mathbb{E}_{P^{T}_{X|Y}}[\boldsymbol{f}_{S_{j}}(X_{T})|Y_{T}=y]\leftarrow\frac{1}{n_{y}}\sum_{i=1}^{N_{T}}\boldsymbol{f}_{S_{j}}(x_{T}^{i})\cdot\mathbb{I}\\{y=y^{i}_{T}\\}$,
$\ \ j=1,\dots,M$

end for

repeat // Optimize weight

$H(\boldsymbol{f}_{T};\boldsymbol{\alpha})\leftarrow
tr(cov(\sum_{j=1}^{M}\alpha_{j}\mathbb{E}_{P^{T}_{X|Y}}[\boldsymbol{f}_{S_{j}}(X_{T})|Y_{T}]))$;

$\boldsymbol{\alpha}\leftarrow\boldsymbol{\alpha}+\lambda\nabla_{\boldsymbol{\alpha}}H(\boldsymbol{f}_{T};\boldsymbol{\alpha})$;

for  _$j\leftarrow 1$ to $M$_ do // Project weight to hyperplane
$\sum_{j=1}^{M}\alpha_{j}=1$

$\alpha_{j}\leftarrow\alpha_{j}-\frac{1}{M}\sum_{j=1}^{M}\alpha_{j}+\frac{1}{M}$

end for

until  _$\alpha$ converges_;

for  _$y\leftarrow 1$ to $d_{T}$_ do // Compute classifier function

$\boldsymbol{g}_{T}(y)=\sum_{j=1}^{M}\
\alpha_{j}\cdot\mathbb{E}_{P_{X|Y}}[\boldsymbol{f}_{S_{j}}(X_{T})|Y_{T}=y]$

end for

Algorithm 1 H-ensemble: Training

### H-Score for Estimating Feature Transferability

Defining the target feature as
$\boldsymbol{f}_{T}=\sum_{j=1}^{M}\alpha_{j}\cdot\boldsymbol{f}_{s_{j}}$, we
introduce the information theoretic metric H-score (Huang et al. 2019b) to
measure the feature transferability for weight optimization. In this section,
we give a detailed formulation of H-score as well as its intuitive
interpretation. From MCA, the mathematical form of H-score is defined as
follows.

######  Definition 1.

With input data $x$, label $y$, feature extractor $\boldsymbol{f}(x)$ and
classifier $\boldsymbol{g}(y)$ (both zero-mean functions). The H-score of
$\boldsymbol{f}$ with regard to the task casting $x$ to $y$ is:

$\displaystyle H(\boldsymbol{f},\boldsymbol{g})=$
$\displaystyle\mathbb{E}_{P_{XY}}\left[\boldsymbol{f}^{\rm
T}(X)\boldsymbol{g}(Y)\right]-\frac{1}{2}tr(cov(\boldsymbol{f}(X))cov(\boldsymbol{g}(Y))).$
(8)

Since we have defined our classifier $\boldsymbol{g}_{T}(y)$ as the optimal
classifier $\boldsymbol{g}_{T}^{*}(y)$ in the previous section, the H-score
can be simplified to the one-sided H-score.

######  Definition 2.

With input data $x$, label $y$ and feature extractor $\boldsymbol{f}(x)$ (a
zero-mean feature function). The one-sided H-score of $\boldsymbol{f}$ with
regard to the task casting $x$ to $y$ is:

$H(\boldsymbol{f})=tr(cov(\boldsymbol{f}(X))^{-1}cov(\mathbb{E}_{P_{X|Y}}[\boldsymbol{f}(X)|Y])).$
(9)

######  Theorem 2.

For a task T with feature extractor $\boldsymbol{f}$ and classifier
$\boldsymbol{g}$, the minimization of $\mathcal{L}$ in maximal correlation
regression is equivalent to the maximization of H-score, i.e.

$\boldsymbol{f}^{*},\boldsymbol{g}^{*}=arg\min_{\boldsymbol{f},\boldsymbol{g}}\mathcal{L}=arg\max_{\boldsymbol{f},\boldsymbol{g}}H(\boldsymbol{f},\boldsymbol{g}).$
(10)

Moreover, if $\boldsymbol{g}$ is defined as the optimal classifier
$\boldsymbol{g}^{*}$, $H(\boldsymbol{f},\boldsymbol{g})$ is then equal to the
one-sided H-score $H(\boldsymbol{f})$, i.e.

$H(\boldsymbol{f},\boldsymbol{g}^{*})=H(\boldsymbol{f}).$ (11)

Theorem 11 indicates that the minimization of
$\mathcal{L}(\boldsymbol{\theta})$ is equivalent to the maximization of the
one-sided H-score $H(\boldsymbol{f})$, and therefore we can determine the
optimal target feature in our method by simply maximizing
$H(\boldsymbol{f}_{T})$.

The formula of one-sided H-score in Eqn. (9) can be intuitively interpreted as
normalizing the inter-class feature variance
$cov(\mathbb{E}_{P_{X|Y}}[\boldsymbol{f}(X)|Y])$ with feature redundancy
$tr(cov(f(X)))$, reflecting the efficiency of extracted feature in
distinguishing different classes. The proof of Eqn. (8), Eqn. (9) and Theorem
11 can be found in (Huang et al. 2019b; Bao et al. 2019; Xu and Huang 2020).
Considering that our target feature $\boldsymbol{f}_{T}$ is normalized, we can
further simplify Eqn. (9) to:

$H(\boldsymbol{f}_{T})=tr(cov(\mathbb{E}_{P_{X|Y}^{T}}[\boldsymbol{f}(X_{T})|Y_{T}])).$
(12)

Eqn. (12) can be explicitly computed from extracted features and labels.
Hence, using target data $D_{T}$, we could efficiently estimate in advance the
transferred performance, or transferability, of target feature
$\boldsymbol{f}_{T}(x)$, and optimize the source weights accordingly. For
clarity, in the following sections we refer to the score defined in Eqn. (12)
as ‘H-score’.

### Source Feature Re-weighting

With the source feature extractors and target samples fixed, the target
feature $\boldsymbol{f}_{T}$ can be viewed as a function of the weight
$\boldsymbol{\alpha}$ defined in Eqn. (4) and therefore the same for its
H-score $H(\boldsymbol{f}_{T})$. Hence, by maximizing
$H(\boldsymbol{f}_{T};\boldsymbol{\alpha})$ with respect to
$\boldsymbol{\alpha}$, we will be able to determine the optimal weight
$\boldsymbol{\alpha}$ for minimizing the loss
$\mathcal{L}(\boldsymbol{\theta})$ in multi-source transfer, alternately
interpreted as deriving the target feature of the highest transferability. We
formulate our optimization problem as follows.

######  Definition 3.

With input data $x_{T}$, label $y_{T}$ and feature extractors
$\boldsymbol{f}_{S_{j}}(x)$ for $j\in\\{1,\dots,M\\}$ (all zero-mean, unit-
variance feature functions), the optimal feature weight
$\boldsymbol{\alpha}=(\alpha_{1},\alpha_{2},\dots,\alpha_{M})^{T}\in\mathbb{R}^{M}$
will be given by:

$\displaystyle\boldsymbol{\alpha}^{*}=arg\max_{\boldsymbol{\alpha}}H\left(\sum_{j=1}^{M}\alpha_{j}\cdot\boldsymbol{f}_{S_{j}}\right)\
s.t.\ \sum_{j=1}^{M}\alpha_{j}=1.$ (13)

The linear constraint of $\boldsymbol{\alpha}$ is obviously convex. We then
verify the benign property of the proposed optimization problem by proving
that the optimal object H-score is also a convex function of
$\boldsymbol{\alpha}$.

######  Theorem 3.

With input data $x$ and label $y$, when weighted $\alpha_{i}$ summing up to 1
and fixed features $\boldsymbol{f}_{i}$ being zero-mean, unit-variance
($i\in\\{1,\dots,n\\}$), the H-score of the weighted feature sum will be a
convex quadratic form of $\boldsymbol{\alpha}$ as below:

$\displaystyle
H(\boldsymbol{f})=H\left(\sum_{i=1}^{n}\alpha_{i}\cdot\boldsymbol{f}_{i}\right)=\sum_{i=1,j=1}^{n,n}\alpha_{i}\alpha_{j}$
(14) $\displaystyle\cdot
tr(\mathbb{E}_{P_{Y}}[\mathbb{E}_{P_{X|Y}}[\boldsymbol{f}_{i}(X)|Y]\cdot\mathbb{E}_{P_{X|Y}}[\boldsymbol{f}_{j}(X)|Y]^{T}]).$

Method | R $\to$ t0 | R $\to$ t1 | R $\to$ t2 | R $\to$ t3 | R $\to$ v0 | R $\to$ v1 | R $\to$ v2 | R $\to$ v3 | Avg.
---|---|---|---|---|---|---|---|---|---
TargetOnly | 0.9730 | 0.9675 | 0.9165 | 0.7260 | 0.9180 | 0.8285 | 0.8750 | 0.7645 | 0.8711
Single-Average | 0.9697 | 0.9659 | 0.9266 | 0.7503 | 0.8416 | 0.7640 | 0.7753 | 0.7013 | 0.8368
Single-Best | 0.9830 | 0.9725 | 0.9405 | 0.7760 | 0.9435 | 0.8470 | 0.8915 | 0.8110 | 0.8956
Average W. | 0.9840 | 0.9770 | 0.9340 | 0.7510 | 0.9240 | 0.8370 | 0.8355 | 0.7910 | 0.8792
MultiFinetune | 0.9695 | 0.9715 | 0.9400 | 0.7570 | 0.9105 | 0.8150 | 0.8240 | 0.7760 | 0.8704
MCW | 0.9890 | 0.9585 | 0.9415 | 0.7675 | 0.9460 | 0.8160 | 0.8655 | 0.7600 | 0.8805
DECISION | 0.5590 | 0.4635 | 0.7700 | 0.3125 | 0.3055 | 0.3245 | 0.5435 | 0.3040 | 0.4478
DATE | 0.2490 | 0.3655 | 0.7740 | 0.2635 | 0.2500 | 0.4545 | 0.5155 | 0.1260 | 0.3748
H-ensemble* | 0.9855 | 0.9745 | 0.9415 | 0.7690 | 0.9470 | 0.8420 | 0.8990 | 0.8300 | 0.8986
Table 3: Accuracy Comparison on VisDA-2017 (10-shot). ‘R’ stands for the rest
tasks. The highest/second-highest accuracy is marked in Bold/Underscore form
respectively. Our method achieves the overall best performance.

A detailed proof of Theorem 14 is presented in the Appendix.I. The
optimization problem of $\boldsymbol{\alpha}$ can therefore be reliably solved
by gradient descent (GD) based algorithms with theoretical guarantee. We then
use the optimal $\boldsymbol{\alpha}$ and derive the optimal target model as
defined in Eqn. (3) with Eqn. (4) and (7).

As a possible solution, we use projected gradient descent (PGD) for
optimization and present the resulting training process of H-ensemble in
Algorithm 1. The derivation of the projection formula in PGD is included in
Appendix.I. The predicted result in testing will be efficiently computed by:

$\displaystyle\hat{y}(x)$
$\displaystyle=arg\max_{y}\left(\sum_{j=1}^{M}\alpha_{j}\boldsymbol{f}_{S_{j}}^{\rm
T}(x)\right)\cdot\boldsymbol{g}_{T}(y).$ (15)

## Experimental Results

### Experiment Setup

#### Datasets.

To verify the effectiveness of H-ensemble in the few-shot MSF setting, we
conduct extensive experiments on four benchmark datasets. VisDA-2017 (Peng et
al. 2017) is a visual domain transfer challenge dataset containing over
280,000 images across 12 categories in training (T, synthetic) and validation
(V, real) domains respectively. Office-31 (Saenko et al. 2010) is a standard
transfer learning dataset with 4,652 images and 31 unbalanced object
categories in three domains: Amazon, DSLR and Webcam, consisting of objects
commonly encountered in office scenarios. Developed from Office-31, Office-
Caltech (Gong et al. 2012) contains 2,533 images covering four domains:
Amazon, DSLR, Webcam and Caltech256, each with 10 categories. Office-Home
(Venkateswara et al. 2017) is a more challenging dataset with 15,599 images in
65 unbalanced categories collected from four domains: Artistic, Clip Art,
Product, and Real-world. 666For an evaluation on large-scale datasets, we also
compared H-ensemble against other methods on DomainNet (Peng et al. 2019)
during rebuttal. The results are included in the Appendix.III.

#### Task Setting.

We split each domain into several tasks. For example, in VisDA-2017 we further
divide both domain V and T into 4 tasks (i.e. v0 - v3 and t0 - t3), each with
3 classes. The resulting eight tasks, encompassing both domain and label
differences, constitute a diverse task pool for the transfer experiment. We do
the same on the Office series datasets and the details can be found in the
Appendix.II.

Following the standard protocol of few-shot learning, the training data for
$k$-shot is $k$ samples per class randomly selected from the target task.
Source models are set to Resnet18 with hidden dim 256 trained on full training
sets. More details of extra dataset, experimental setup and model
implementation are discussed in the Appendix.II.

### Comparison with Baselines

#### Baselines.

For a general performance evaluation, we conduct comparison experiments using
the first task synthesis strategy. Considering very few methods are designed
for few-shot MSF transfer setting, we take SOTA works under similar settings
and variations of our method as baselines. The compared methods include: 1)
Trivial Solutions: Target Only, Single-Best, Single-Avg.; 2) Relevant
Approaches: Average W., MultiFinetune; 3) Prior SOTAs: MCW (Lee, Sattigeri,
and Wornell 2019), DECISION (Ahmed et al. 2021), DATE (Han et al. 2023). Here,
Single-* adapt one source model to target task using MCR and *-Best/Avg. stand
for the best/average performance. Target Only finetunes Imagenet-pretrained
model on few-shot target data only. Average W. replace the weighted sum in our
framework with equal weights. MultiFinetune learns a fully connected
classifier with the target extractor learned in H-ensemble. The Prior SOTAs
are the SOTAs with the closest problem setting (MSFDA).

#### Results and Analysis.

We take VisDA-2017 10-shot as an example777Experiments on other datasets &
shots and statistical results are listed in Appendix.III. and summarize the
results in Table 3. Overall, H-ensemble performs the best among all the
methods including the best single source transfer, especially in the more
challenging real world image (V) domain. Here the significant degradation of
DECISION and DATE is due to the change of problem setting from UDA to FS
transfer.

Figure 3: Source Weights derived in Different Methods. (VisDA-2017, target v3,
10-shot) The source weight in H-ensemble (red) conforms most to intuition,
emphasizing on tasks from the same domain (v0, v1, v2).

For further analysis, we plot the source weights for task v3 derived in
weighting-based methods in Fig. 3. It is clearly illustrated that given four
tasks on domain V and T each, H-ensemble can successfully recognize the
importance of integrating tasks from the same domain (v0, v1, v2) and attach
them with high weights. It also correctly emphasizes on t3 (the only task
sharing the same class labels with v3) the most among tasks on domain T,
displaying an outstanding and interpretable ability to measure source
importance.

### Ablation Study

We further explore the capability of H-ensemble when transferring to harder
tasks on Office datasets, as well as verify components in H-ensemble by
ablating. The baselines are constructed by substituting each part with naive
solutions. Specifically, we have:

  * •

Single MCR: multi-source $\to$ average performance of single-source transfer;

  * •

Average W.: weighting strategy $\to$ average weights;

  * •

MultiFinetune: MCR classifier $\to$ finetune classifier.

For each variety, we record the average transfer performance on each dataset,
concluding the results of 8-shot888Compared to VisDA-2017, the shot reduces
due to fewer samples per class in Office-* datasets. More results in
Appendix.III. in Table 4, where both the effectiveness of H-ensemble and the
necessity of every module are explicitly supported. It is also notable that
our whole framework is established on a united theory ground, and any
substitution may break its theoretical reliability and interpretability. We
also visualize the target feature extracted by random source, average weight
and H-ensemble in Fig. 4, showing that H-ensemble extracts features with the
strongest class discriminability for the given task.

Method | Office-31 | Office-Caltech | Office-Home
---|---|---|---
Single MCR | 0.8483 | 0.9645 | 0.6235
Average W. | 0.8479 | 0.9716 | 0.6702
MultiFinetune | 0.8414 | 0.9663 | 0.6571
H-ensemble* | 0.8644 | 0.9708 | 0.6940
Table 4: Ablation Study on Office Datasets (8-shot). Our weighting strategy
and optimal MCR classifier both contribute to the overall effectiveness.

Figure 4: Feature extracted by different models, visualized by t-SNE. From
left to right: random source, Average W. and H-ensemble. It turns out that
features generated by our method have the greatest discriminability.

## Conclusion

In this work, we highlight a common yet largely underexplored scenario of
multi-source transfer which we refer to as few-shot MSF transfer, and take the
lead in giving a mathematical formulation of it. Addressing this problem, we
take on an information theoretic perspective and propose our H-ensemble
framework. Consisting of three main components, H-ensemble dynamically learns
the optimal linear ensemble of source models for the target task, with the
ensemble weights optimized by maximizing H-score and classifier determined by
a generalization of MCR. We present detailed theoretic deduction and
interpretation as well as extensive experimental validation for our method,
showing that H-ensemble can effectively boost the learning on the target task
in the few-shot MSF transfer scheme.

## Acknowledgement

This study is supported in part by the Tsinghua SIGS Scientific Research
Start-up Fund (Grant QD2021012C), Natural Science Foundation of China (Grant
62001266) and Shenzhen Key Laboratory of Ubiquitous Data Enabling (No.
ZDSYS20220527171406015).

## References

  * Agostinelli et al. (2022) Agostinelli, A.; Uijlings, J.; Mensink, T.; and Ferrari, V. 2022.  Transferability metrics for selecting source model ensembles.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7936–7946.
  * Ahmed et al. (2021) Ahmed, S. M.; Raychaudhuri, D. S.; Paul, S.; Oymak, S.; and Roy-Chowdhury, A. K. 2021.  Unsupervised multi-source domain adaptation without access to source data.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 10103–10112.
  * Bao et al. (2019) Bao, Y.; Li, Y.; Huang, S.-L.; Zhang, L.; Zheng, L.; Zamir, A.; and Guibas, L. 2019.  An information-theoretic approach to transferability in task transfer learning.  In _2019 IEEE international conference on image processing (ICIP)_ , 2309–2313. IEEE.
  * Ben-David et al. (2010) Ben-David, S.; Blitzer, J.; Crammer, K.; Kulesza, A.; Pereira, F.; and Vaughan, J. W. 2010.  A theory of learning from different domains.  _Machine learning_ , 79: 151–175.
  * Chen and Marchand (2023) Chen, Q.; and Marchand, M. 2023.  Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation.  In _International Conference on Artificial Intelligence and Statistics_ , 10368–10394. PMLR.
  * Christodoulidis et al. (2016) Christodoulidis, S.; Anthimopoulos, M.; Ebner, L.; Christe, A.; and Mougiakakou, S. 2016.  Multisource transfer learning with convolutional neural networks for lung pattern analysis.  _IEEE journal of biomedical and health informatics_ , 21(1): 76–84.
  * Crammer, Kearns, and Wortman (2008) Crammer, K.; Kearns, M.; and Wortman, J. 2008.  Learning from Multiple Sources.  _Journal of Machine Learning Research_ , 9(8).
  * Dong et al. (2021) Dong, J.; Fang, Z.; Liu, A.; Sun, G.; and Liu, T. 2021.  Confident anchor-induced multi-source free domain adaptation.  _Advances in Neural Information Processing Systems_ , 34: 2848–2860.
  * Fang et al. (2019) Fang, W.; Chen, C.; Song, B.; Wang, L.; Zhou, J.; and Zhu, K. Q. 2019.  Adapted tree boosting for transfer learning.  In _2019 IEEE International Conference on Big Data (Big Data)_ , 741–750. IEEE.
  * Fang et al. (2022) Fang, Y.; Yap, P.-T.; Lin, W.; Zhu, H.; and Liu, M. 2022.  Source-free unsupervised domain adaptation: A survey.  _arXiv preprint arXiv:2301.00265_.
  * Feller (1991) Feller, W. 1991.  _An introduction to probability theory and its applications, Volume 2_ , volume 81.  John Wiley & Sons.
  * Feng et al. (2021) Feng, H.; You, Z.; Chen, M.; Zhang, T.; Zhu, M.; Wu, F.; Wu, C.; and Chen, W. 2021.  KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation.  In _ICML_ , 3274–3283.
  * Gebelein (1941) Gebelein, H. 1941.  Das statistische Problem der Korrelation als Variations-und Eigenwertproblem und sein Zusammenhang mit der Ausgleichsrechnung.  _ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik_ , 21(6): 364–379.
  * Gong et al. (2012) Gong, B.; Shi, Y.; Sha, F.; and Grauman, K. 2012.  Geodesic flow kernel for unsupervised domain adaptation.  In _2012 IEEE conference on computer vision and pattern recognition_ , 2066–2073. IEEE.
  * Han et al. (2023) Han, Z.; Zhang, Z.; Wang, F.; He, R.; Su, W.; Xi, X.; and Yin, Y. 2023.  Discriminability and Transferability Estimation: A Bayesian Source Importance Estimation Approach for Multi-Source-Free Domain Adaptation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, 7811–7820.
  * Hirschfeld (1935) Hirschfeld, H. O. 1935.  A connection between correlation and contingency.  In _Mathematical Proceedings of the Cambridge Philosophical Society_ , volume 31, 520–524. Cambridge University Press.
  * Hoffman et al. (2018) Hoffman, J.; Tzeng, E.; Park, T.; Zhu, J.-Y.; Isola, P.; Saenko, K.; Efros, A.; and Darrell, T. 2018.  Cycada: Cycle-consistent adversarial domain adaptation.  In _International conference on machine learning_ , 1989–1998. Pmlr.
  * Huang, Wang, and Qin (2012) Huang, P.; Wang, G.; and Qin, S. 2012.  Boosting for transfer learning from multiple data sources.  _Pattern Recognition Letters_ , 33(5): 568–579.
  * Huang et al. (2019a) Huang, S.-L.; Makur, A.; Wornell, G. W.; and Zheng, L. 2019a.  On universal features for high-dimensional learning and inference.  _arXiv preprint arXiv:1911.09105_.
  * Huang et al. (2019b) Huang, S.-L.; Xu, X.; Zheng, L.; and Wornell, G. W. 2019b.  An information theoretic interpretation to deep neural networks.  In _2019 IEEE international symposium on information theory (ISIT)_ , 1984–1988. IEEE.
  * Ibrahim, Ponomareva, and Mazumder (2022) Ibrahim, S.; Ponomareva, N.; and Mazumder, R. 2022.  Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , 693–709. Springer.
  * Lee, Sattigeri, and Wornell (2019) Lee, J.; Sattigeri, P.; and Wornell, G. 2019.  Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks.  _Advances in neural information processing systems_ , 32.
  * Li et al. (2022) Li, K.; Lu, J.; Zuo, H.; and Zhang, G. 2022.  Source-free multi-domain adaptation with generally auxiliary model training.  In _2022 International Joint Conference on Neural Networks (IJCNN)_ , 1–8. IEEE.
  * Mansour, Mohri, and Rostamizadeh (2008) Mansour, Y.; Mohri, M.; and Rostamizadeh, A. 2008.  Domain adaptation with multiple sources.  _Advances in neural information processing systems_ , 21.
  * Peng et al. (2019) Peng, X.; Bai, Q.; Xia, X.; Huang, Z.; Saenko, K.; and Wang, B. 2019.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 1406–1415.
  * Peng et al. (2017) Peng, X.; Usman, B.; Kaushik, N.; Hoffman, J.; Wang, D.; and Saenko, K. 2017.  Visda: The visual domain adaptation challenge.  _arXiv preprint arXiv:1710.06924_.
  * Perkins, Salomon et al. (1992) Perkins, D. N.; Salomon, G.; et al. 1992.  Transfer of learning.  _International encyclopedia of education_ , 2: 6452–6457.
  * Rényi (1959) Rényi, A. 1959.  On measures of dependence.  _Acta mathematica hungarica_ , 10(3-4): 441–451.
  * Saenko et al. (2010) Saenko, K.; Kulis, B.; Fritz, M.; and Darrell, T. 2010.  Adapting visual category models to new domains.  In _Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11_ , 213–226. Springer.
  * Shui et al. (2021) Shui, C.; Li, Z.; Li, J.; Gagné, C.; Ling, C. X.; and Wang, B. 2021.  Aggregating from multiple target-shifted sources.  In _International Conference on Machine Learning_ , 9638–9648. PMLR.
  * Sun et al. (2011) Sun, Q.; Chattopadhyay, R.; Panchanathan, S.; and Ye, J. 2011.  A two-stage weighting framework for multi-source domain adaptation.  _Advances in neural information processing systems_ , 24.
  * Sun, Shi, and Wu (2015) Sun, S.; Shi, H.; and Wu, Y. 2015.  A survey of multi-source domain adaptation.  _Information Fusion_ , 24: 84–92.
  * Tong et al. (2021) Tong, X.; Xu, X.; Huang, S.-L.; and Zheng, L. 2021.  A mathematical framework for quantifying transferability in multi-source transfer learning.  _Advances in Neural Information Processing Systems_ , 34: 26103–26116.
  * Venkateswara et al. (2017) Venkateswara, H.; Eusebio, J.; Chakraborty, S.; and Panchanathan, S. 2017.  Deep hashing network for unsupervised domain adaptation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 5018–5027.
  * Wang et al. (2020) Wang, Y.; Yao, Q.; Kwok, J. T.; and Ni, L. M. 2020.  Generalizing from a few examples: A survey on few-shot learning.  _ACM computing surveys (csur)_ , 53(3): 1–34.
  * Weiss, Khoshgoftaar, and Wang (2016) Weiss, K.; Khoshgoftaar, T. M.; and Wang, D. 2016.  A survey of transfer learning.  _Journal of Big data_ , 3(1): 1–40.
  * Xu and Huang (2020) Xu, X.; and Huang, S.-L. 2020.  Maximal correlation regression.  _IEEE Access_ , 8: 26591–26601.
  * Xu and Sun (2012) Xu, Z.; and Sun, S. 2012.  Multi-source transfer learning with multi-view adaboost.  In _Neural Information Processing: 19th International Conference, ICONIP 2012, Doha, Qatar, November 12-15, 2012, Proceedings, Part III 19_ , 332–339. Springer.
  * Yang et al. (2023) Yang, Y.; Wei, M.; He, J.; Yang, J.; Ye, J.; and Gu, Y. 2023.  Pick the Best Pre-trained Model: Towards Transferability Estimation for Medical Image Segmentation.  _arXiv preprint arXiv:2307.11958_.
  * Zhuang et al. (2020) Zhuang, F.; Qi, Z.; Duan, K.; Xi, D.; Zhu, Y.; Zhu, H.; Xiong, H.; and He, Q. 2020.  A comprehensive survey on transfer learning.  _Proceedings of the IEEE_ , 109(1): 43–76.