# Episodic Multi-Task Learning with
Heterogeneous Neural Processes

Jiayi Shen1, Xiantong Zhen1,2 , Qi (Cheems) Wang3, Marcel Worring1
1University of Amsterdam, Netherlands, {j.shen<EMAIL_ADDRESS>
2 Inception Institute of Artificial Intelligence, Abu Dhabi, UAE,
<EMAIL_ADDRESS>
3 Kaiyuan Mathematical Sciences Institute, Changsha, China<EMAIL_ADDRESS>Currently with United Imaging Healthcare, Co., Ltd., China.

###### Abstract

This paper focuses on the data-insufficiency problem in multi-task learning
within an episodic training setup. Specifically, we explore the potential of
heterogeneous information across tasks and meta-knowledge among episodes to
effectively tackle each task with limited data. Existing meta-learning methods
often fail to take advantage of crucial heterogeneous information in a single
episode, while multi-task learning models neglect reusing experience from
earlier episodes. To address the problem of insufficient data, we develop
Heterogeneous Neural Processes (HNPs) for the episodic multi-task setup.
Within the framework of hierarchical Bayes, HNPs effectively capitalize on
prior experiences as meta-knowledge and capture task-relatedness among
heterogeneous tasks, mitigating data-insufficiency. Meanwhile, transformer-
structured inference modules are designed to enable efficient inferences
toward meta-knowledge and task-relatedness. In this way, HNPs can learn more
powerful functional priors for adapting to novel heterogeneous tasks in each
meta-test episode. Experimental results show the superior performance of the
proposed HNPs over typical baselines, and ablation studies verify the
effectiveness of the designed inference modules.

##  1 Introduction

Deep learning models have made remarkable progress with the help of the
exponential increase in the amount of available training data [1]. However,
many practical scenarios only have access to limited labeled data [2]. Such
data-insufficiency sharply degrades the model’s performance [2, 3]. Both meta-
learning and multi-task learning have the potential to alleviate the data-
insufficiency issue. Meta-learning can extract meta-knowledge from past
episodes and thus enables rapid adaptation to new episodes with a few examples
only [4, 5, 6, 7]. Meanwhile, multi-task learning exploits the correlation
among several tasks and results in more accurate learners for all tasks
simultaneously [8, 9, 10, 11]. However, the integration of meta-learning and
multi-task learning in overcoming the data-insufficiency problem is rarely
investigated.

In episodic training [4], existing meta-learning methods [4, 5, 6, 7, 12, 13]
in every meta-training or meta-test episode learn a single-task. In this
paper, we refer to this conventional setting as episodic single-task learning.
This setting restricts the potential for these models to explore task-
relatedness within each episode, leaving the learning of multiple
heterogeneous tasks in a single episode under-explored. We consider multiple
tasks in each episode as episodic multi-task learning. The crux of episodic
multi-task learning is to generalize the ability of exploring task-relatedness
from meta-training to meta-test episodes. The differences between episodic
single-task learning and episodic multi-task learning are illustrated in
Figure 1. To be specific, we restrict the scope of the problem setup to the
case where tasks in each meta-training or meta-test episode are heterogeneous
but also relate to each other by sharing the same target space.

Figure 1: Illustration of episodic multi-task learning. Each row corresponds
to a meta-training or meta-test episode. Different colors represent different
label spaces among episodes; the same color with different shades represents
different categories in the same task. Compared with episodic single-task
learning, episodic multi-task learning simultaneously handles several related
tasks in a single episode.

The neural process (NP) family [12, 13], as typical meta-learning
probabilistic models [14], efficiently quantifies predictive uncertainty with
limited data, making it in principle well-suited for tackling the problem of
data-insufficiency. However, in practice, it is challenging for vanilla NPs
[12] with a global latent variable to encode beneficial heterogeneous
information in each episode. This issue is also known as the expressiveness
bottleneck [15, 16], which weakens the model’s capacity to handle insufficient
data, especially when faced with diverse heterogeneous tasks.

To better resolve the data-insufficiency problem, we develop Heterogeneous
Neural Processes (HNPs) for episodic multi-task learning. As a new member of
the NP family, HNPs improve the expressiveness of vanilla NPs by introducing a
hierarchical functional space with global and local latent variables. The
remainder of this work is structured as follows: We introduce our method in
Section (2). Related work is overviewed in Section (3). We report experimental
results with analysis in Section (4), after which we conclude with a technical
discussion, existing limitations, and future extensions. In detail, our
technical contributions are two-fold:

  * •

Built on the hierarchical Bayes framework, our developed HNPs can
simultaneously generalize meta-knowledge from past episodes to new episodes
and exploit task-relatedness across heterogeneous tasks in every single
episode. This mechanism makes HNPs more powerful when encoding complex
conditions into functional priors.

  * •

We design transformer-structured inference modules to infer the hierarchical
latent variables, capture task-relatedness, and learn a set of tokens as meta-
knowledge. The designed modules can fuse the meta-knowledge and heterogeneous
information from context samples in a unified manner, boosting the
generalization capability of HNPs across tasks and episodes.

Experimental results show that the proposed HNPs together with transformer-
structured inference modules, can exhibit superior performance on regression
and classification tasks under the episodic multi-task setup.

##  2 Methodology

Notations 111For ease of presentation, we abbreviate a set
$\\{{(\cdot)}^{m}\\}^{M}_{m=1}$ as ${(\cdot)}^{1:M}$, where $M$ is a positive
integer. Likewise, $\\{{(\cdot)}_{o}\\}^{O}_{o=1}$ is abbreviated as
${(\cdot)}_{1:O}$. For convenience, the notation table is provided in Appendix
B.. We will now formally define episodic multi-task learning. For a single
episode $\tau$, we consider $M$ heterogeneous but related tasks
$\mathcal{I}^{1:M}_{\tau}=\\{\mathcal{I}^{m}_{\tau}\\}_{m=1}^{M}$. Notably,
the subscript denotes an episode, while superscripts are used to distinguish
tasks in this episode. In the episodic multi-task setup, tasks in a single
episode are heterogeneous since they are sampled from different task
distributions $\\{p(\mathcal{I}^{m})\\}_{m=1}^{M}$, but are related at the
same time as they share the target space $\mathcal{Y}_{\tau}$.

To clearly relate to the modeling of vanilla neural processes [12], this paper
follows its nomenclature to define each task. Note that in vanilla neural
processes context and target are often respectively called support and query
in conventional meta-learning [4, 5]. Each task $\mathcal{I}^{m}_{\tau}$
contains a context set with limited training data
$\mathcal{C}_{\tau}^{m}=\\{{\bar{x}}_{\tau,i}^{m},{\bar{y}}_{\tau,i}^{m}\\}_{i=1}^{N_{\mathcal{C}}}$
and a target set
$\mathcal{T}_{\tau}^{m}=\\{{{x}}_{\tau,j}^{m},{{y}}_{\tau,j}^{m}\\}_{j=1}^{N_{\mathcal{T}}}$,
where $N_{\mathcal{C}}$ and $N_{\mathcal{T}}$ are the numbers of context
samples and target samples, respectively. $\bar{{x}}_{\tau,i}^{m}$ and
${x}_{\tau,j}^{m}$ represent features of context and target samples; while
$\bar{{y}}_{\tau,i}^{m},{y}_{\tau,j}^{m}\in\mathcal{Y}_{\tau}$ are their
corresponding targets, where
$i=1,2,...,N_{\mathcal{C}};j=1,2,..,N_{\mathcal{T}};m=1,2,...,M$. For
simplicity, we denote the set of target samples and their corresponding
ground-truths by
$\mathbf{x}_{\tau}^{m}=\\{{x}_{\tau,j}^{m}\\}_{j=1}^{N_{\mathcal{T}}}$,
$\mathbf{y}_{\tau}^{m}=\\{{y}_{\tau,j}^{m}\\}_{j=1}^{N_{\mathcal{T}}}$. For an
episode $\tau$, episodic multi-task learning aims to perform simultaneously
well on each corresponding target set $\mathcal{T}_{\tau}^{m},m=1,2..,M$,
given the collection of context sets $\mathcal{C}_{\tau}^{1:M}$.

For classification, this paper follows the protocol of meta models [4, 5, 17],
such as $O$-way $K$-shot setup, clearly suffering from the data-insufficiency
problem. Thus, episodic multi-task classification can be cast as a $M$-task
$O$-way $K$-shot supervised learning problem. An episode has $M$ related
classification tasks, and each of them has a context set with $K$ different
instances from each of the $O$ classes [5]. It is worth mentioning that the
target spaces of meta-training episodes do not overlap with any categories in
those of meta-test episodes.

###  2.1 Modeling and Inference of Heterogeneous Neural Processes

We now present the proposed heterogeneous neural process. The proposed model
inherits the advantages of multi-task learning and meta-learning, which can
exploit task-relatedness among heterogeneous tasks and extract meta-knowledge
from previous episodes. Next, we characterize the generative process, clarify
the modeling within the hierarchical Bayes framework, and derive the
approximate evidence lower bound (ELBO) in optimization.

#### Generative Processes.

To get to our proposed method HNPs, we extend the distribution over a single
function $p(f_{\tau})$ as used in vanilla NPs to a joint distribution of
multiple functions $p(f^{1:M}_{\tau})$ for all heterogeneous tasks in a single
episode $\tau$. In detail, the underlying multi-task function distribution
$p(f^{1:M}_{\tau})$ is inferred from a collection of context sets
$\mathcal{C}^{1:M}_{\tau}$ and learnable meta-knowledge $\omega,\nu^{1:M}$.
Note that $\omega$ represents the shared meta-knowledge for all tasks, and
$\nu^{m}$ denotes the task-specific meta-knowledge corresponding to the task
distribution $p(\mathcal{I}^{m})$. Hence, we can formulate the predictive
distribution for every single episode as follows:

$\displaystyle
p(\mathcal{T}_{\tau}^{1:M}|\mathcal{C}_{\tau}^{1:M};\omega,\nu^{1:M})=\int
p(\mathbf{y}_{\tau}^{1:M}|\mathbf{x}_{\tau}^{1:M},f_{\tau}^{1:M})p(f_{\tau}^{1:M}|\mathcal{C}_{\tau}^{1:M};\omega,\nu^{1:M})df_{\tau}^{1:M},$
(1)

where $p(f_{\tau}^{1:M}|\mathcal{C}_{\tau}^{1:M};\omega,\nu^{m})$ denotes the
data-dependent functional prior for multiple tasks of the episode $\tau$. The
functional prior encodes context sets from all heterogeneous tasks and
quantifies uncertainty in the functional space. Nevertheless, it is less
optimal to characterize multi-task function generative processes with vanilla
NPs, since the single latent variable limits the capacity of the latent space
to specify the complicated functional priors. This expressiveness bottleneck
in vanilla NPs is particularly severe for our episodic multi-task learning
since each episode has diverse heterogeneous tasks with insufficient data.

Figure 2: Graphical model of the proposed HNPs in a single episode. Filled
shapes indicate observations. Probabilistic and deterministic variables are
indicated by unfilled circles and diamonds, respectively.

#### Modeling within the Hierarchical Bayes Framework.

To mitigate the expressiveness bottleneck of vanilla NPs, we model HNPs by
parameterizing each task-specific function within a hierarchical Bayes
framework. As illustrated in Figure 2, HNPs integrate a global latent
representation $\mathbf{z}_{\tau}^{m}$ and a set of local latent parameters
$\mathbf{w}_{\tau,1:O}^{m}$ to model each task-specific function
$f_{\tau}^{m}$. Specifically, the latent variables are introduced at different
levels: $\mathbf{z}_{\tau}^{m}$ encodes task-specific context information from
$\mathcal{C}_{\tau}^{m}$ and $\nu^{m}$ in the representation level.
$\mathbf{w}_{\tau,1:O}^{m}$ encode prediction-aware information for a task-
specific decoder from $\mathcal{C}_{\tau}^{1:M}$ and $\omega$ in the parameter
level, where $O$ is the dimension of the decoder. For example, the dimension
is the size of the target space when performing classification tasks.

Notably, each local latent parameter is conditioned on the global latent
representation, which controls access to all context sets in the episode for
the corresponding task. Our method differs from previous hierarchical
architectures [16, 18, 19, 20] in the NP family since the local latent
parameters of our HNPs are prediction-aware and explicitly constitute a
decoder for the subsequent inference processes.

In practice, we assume that distributions of each task-specific function are
conditionally independent. Thus, with the introduced hierarchical latent
variables for each task in the episode, we can factorize the prior
distribution over multiple functions in Eq. (1) into:

$\displaystyle
p(f_{\tau}^{1:M}|\mathcal{C}_{\tau}^{1:M};\omega,\nu^{1:M})=\prod_{m=1}^{M}p(\mathbf{z}_{\tau}^{m}|\mathcal{C}_{\tau}^{m};\nu^{m})p(\mathbf{w}_{\tau,1:O}^{m}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega),$
(2)

where $p(\mathbf{z}_{\tau}^{m}|\mathcal{C}_{\tau}^{m};\nu^{m})$ and
$p(\mathbf{w}_{\tau,1:O}^{m}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega)$
are prior distributions of the global latent representation and the local
latent parameters to induce the task-specific function distribution.

By integrating Eq. (2) into Eq. (1), we rewrite the modeling of HNPs in the
following form:

$\displaystyle
p(\mathcal{T}_{\tau}^{1:M}|\mathcal{C}_{\tau}^{1:M};\omega,\nu^{1:M})=$
$\displaystyle\prod_{m=1}^{M}\int\Big{\\{}\int
p(\mathbf{y}_{\tau}^{m}|\mathbf{x}_{\tau}^{m},\mathbf{w}_{\tau,1:O}^{m})$ (3)
$\displaystyle
p(\mathbf{w}_{\tau,1:O}^{m}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega)d\mathbf{w}^{m}_{\tau,1:O}\Big{\\}}p(\mathbf{z}_{\tau}^{m}|\mathcal{C}_{\tau}^{m};\nu^{m})d\textbf{z}_{\tau}^{m},$

where
$p(\mathbf{y}_{\tau}^{m}|\mathbf{x}_{\tau}^{m},\mathbf{w}_{\tau,1:O}^{m})$ is
the function distribution for the task $\mathcal{I}^{m}_{\tau}$ in HNPs. This
distribution is obtained by the matrix multiplication of
$\mathbf{x}_{\tau}^{m}$ and all local latent parameters
$\mathbf{w}_{\tau,1:O}^{m}$.

Compared with most NP models [12, 16, 18, 19] employing only latent
representations, HNPs infer both latent representations and parameters in the
hierarchical architecture from multiple heterogeneous context sets and
learnable meta-knowledge. Our model specifies a richer and more intricate
functional space by leveraging the hierarchical uncertainty inherent in the
context sets and meta-knowledge. This theoretically yields more powerful
functional priors to induce multi-task function distributions.

Moreover, we claim that the developed model constitutes an _exchangeable
stochastic process_ and demonstrate this via Kolmogorov Extension Theorem
[21]. Please refer to Appendix B for the proof.

#### Approximate ELBO.

Since both exact functional posteriors and priors are intractable, we apply
variational inference to the proposed HNPs in Eq. (3). This results in the
approximate ELBO:

$\displaystyle
L_{\rm{HNPs}}(\omega,\nu^{1:M},\theta,\phi)=\sum_{m=1}^{M}\bigg{\\{}\mathbb{E}_{q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})}\Big{\\{}\mathbb{E}_{q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega)}[\log
p(\mathbf{y}_{\tau}^{m}|\mathbf{x}_{\tau}^{m},\mathbf{w}^{m}_{\tau,1:O})]$ (4)
$\displaystyle-\mathbb{D}_{\rm{KL}}[q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega)||p_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega)]\Big{\\}}-\mathbb{D}_{\rm{KL}}[q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})||p_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{C}_{\tau}^{m};\nu^{m})]\bigg{\\}},$

where $q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})$ and
$q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega)$
are variational posteriors of their corresponding latent variables. $\theta$
and $\phi$ are parameters of inference modules for $\mathbf{z}^{m}_{\tau}$ and
$\mathbf{w}^{m}_{\tau,1:O}$, respectively. Following the protocol of vanilla
NPs [12], the priors use the same inference modules as variational posteriors
for tractable optimization. In this way, the KL-divergence terms in Eq. (4)
encourage all latent variables inferred from the context sets to stay close to
those inferred from the target sets, enabling effective function generation
with few examples. Details on the derivation of the approximate ELBO and its
tractable optimization are attached in Appendix C.

Figure 3: A diagram of transformer-structured inference modules of HNPs for
the first meta-training episode in Figure 1 under the 3-task 5-way 1-shot
setting. For clarity, we display the inference process of the local latent
parameters specific to the third task in the episode.

###  2.2 Transformer-Structured Inference Module

In order to infer the prior and variational posterior distributions in Eq.
(4), it is essential to develop well-designed approximate inference modules.
This is non-trivial and closely related to the performance of HNPs. Here we
adopt a transformer structure as the inference module to better exploit task-
relatedness from the meta-knowledge and the context sets in the episode. More
specifically, the previously mentioned meta-knowledge $\omega=\omega_{1:O}$
and $\nu^{1:M}$ are instantiated as learnable tokens to induce the
distributions of hierarchical latent variables in the proposed model.

Without loss of generality, in the next, we provide an example of transformer-
structured inference modules for prior distributions in classification
scenarios. Details of the inference modules in regression scenarios can be
found in Appendix D. In Figure 3, a diagram of the transformer-structured
inference modules is displayed under the 3-task 5-way 1-shot setting. In this
case, the number of context samples is the same as the size of the target
space, and thus we have
$\mathcal{C}_{\tau}^{m}=\\{\bar{x}^{m}_{\tau,o},\bar{y}^{m}_{\tau,o}\\}^{O}_{o=1}$,
where $O$ is set as 5. In episodic training, labels in context sets are always
available during inference.

#### Transformer-Structured Inference Module $\\{\theta,\nu^{m}\\}$ for
$\mathbf{z}_{\tau}^{m}$.

In the proposed HNPs, each global latent representation encodes task-specific
information relevant to the considered task in the episode as
$p_{\theta}(\mathbf{z}^{m}_{\tau}|\mathcal{C}_{\tau}^{m};\nu^{m})$. The
learnable token $\nu^{m}$ preserves the meta-knowledge from previous episodes
for specific tasks, which are sampled from the corresponding task distribution
$p(\mathcal{I}^{m})$. The role of $\nu^{m}$ is to help the model adapt
efficiently to such specific tasks in meta-test episodes.

In detail, we set the dimension of the learnable token $\nu^{m}$ to the same
as that of the features $\bar{x}^{m}_{\tau,1:O}$. Then the transformer-
structured inference module $\theta$ fuses them in a unified manner by taking
$[\bar{x}^{m}_{\tau,1:O};\nu^{m}]$ as the input. The module $\theta$ outputs
the mean and variance of the corresponding prior distribution. The inference
steps for the global latent representation $\mathbf{z}^{m}_{\tau}$ are:

$[\widetilde{x}^{m}_{\tau,1:O};\widetilde{\nu}^{m}]=\texttt{MSA}(\texttt{LN}([\bar{x}^{m}_{\tau,1:O};\nu^{m}]))+[\bar{x}^{m}_{\tau,1:O};\nu^{m}],$
(5)
$[\widehat{x}^{m}_{\tau,1:O};\widehat{\nu}^{m}]=\texttt{MLP}(\texttt{LN}([\widetilde{x}^{m}_{\tau,1:O};\widetilde{\nu}^{m}]))+[\widetilde{x}^{m}_{\tau,1:O};\widetilde{\nu}^{m}],$
(6)
$p_{\theta}(\mathbf{z}^{m}_{\tau}|\mathcal{C}_{\tau}^{m};\nu^{m})=\mathcal{N}(\mathbf{z}^{m}_{\tau};\mu_{\mathbf{z}^{m}_{\tau}},\sigma_{\mathbf{z}^{m}_{\tau}}),$
(7)

where
$\mu_{\mathbf{z}^{m}_{\tau}}=\texttt{MLP}(\widehat{\nu}^{m}),\sigma_{\mathbf{z}^{m}_{\tau}}=\texttt{Softplus}(\texttt{MLP}(\widehat{\nu}^{m}))$.
The transformer-structured inference module includes a multi-headed self-
attention (MSA) and three multi-layer perceptrons (MLP). The layer
normalization (LN) is "pre-norm" as done in [22]. Softplus is the activation
function to output the appropriate value as the variance of the prior
distribution [23].

#### Transformer-Structured Inference Module $\\{\phi,\omega_{1:O}\\}$ for
$\mathbf{w}_{\tau,1:O}^{m}$.

Likewise, each learnable token $\omega_{o}$ corresponds to a local latent
parameter $\mathbf{w}_{\tau,o}^{m}$. With the learnable tokens $\omega_{1:O}$,
we reformulate the prior distribution of local latent parameters as
$p_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega_{1:O})$.
In this way, we learn the shared knowledge, inductive biases across all tasks,
and their distribution at a parameter level, which in practical settings can
capture epistemic uncertainty.

To be specific, the prior distribution can be factorized as
$\prod_{o=1}^{O}p_{\phi}(\mathbf{w}^{m}_{\tau,o}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega_{o})$,
where all local latent parameters are assumed to be conditionally independent.
For each local latent parameter $\mathbf{w}^{m}_{\tau,o}$, the transformer-
structured inference module $\phi$ takes $[\bar{x}^{1:M}_{\tau,o},\omega_{o}]$
as input and outputs the corresponding prior distribution, where
$\bar{x}^{1:M}_{\tau,o}$ are deep features from the same class $o$ in the
episode and $\omega_{o}$ is the corresponding learnable token. Here the
inference steps for $\mathbf{w}^{m}_{\tau,o}$ are as follows:

$[\widetilde{x}^{1:M}_{\tau,o};\widetilde{\omega}_{o}]=\texttt{MSA}(\texttt{LN}([\bar{x}^{1:M}_{\tau,o};\omega_{o}]))+[\bar{x}^{1:M}_{\tau,o};\omega_{o}],$
(8)
$[\widehat{x}^{1:M}_{\tau,o};\widehat{\omega}_{o}]=\texttt{MLP}(\texttt{LN}([\widetilde{x}^{1:M}_{\tau,o};\widetilde{\omega}_{o}]))+[\widetilde{x}^{1:M}_{\tau,o};\widetilde{\omega}_{o}],$
(9) $\displaystyle
p_{\phi}(\mathbf{w}^{m}_{\tau,o}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega_{o})=\mathcal{N}(\mathbf{w}^{m}_{\tau,o};\mu_{\mathbf{w}^{m}_{\tau,o}},\sigma_{\mathbf{w}^{m}_{\tau,o}}),$
(10)

where
$\mu_{\mathbf{w}^{m}_{\tau,o}}=\texttt{MLP}(\widehat{\omega}_{o},{\mathbf{z}^{m}_{\tau}}^{(i)}),\sigma_{\mathbf{w}^{m}_{\tau,o}}=\texttt{Softplus}(\texttt{MLP}(\widehat{\omega}_{o},{\mathbf{z}^{m}_{\tau}}^{(i)}))$.
${\mathbf{z}^{m}_{\tau}}^{(i)}$ is a Monte Carlo sample from the variational
posterior of the corresponding global latent representation during meta-
training.

Both transformer-structured inference modules use the refined tokens
$\widehat{\nu}^{m}$ and $\widehat{\omega}_{o}$ to obtain a global latent
representation and a local latent parameter, respectively. The introduced
tokens preserve the specific meta-knowledge for each latent variable during
inference. Compared with the $\theta$-parameterised inference module exploring
the intra-task relationships, the $\phi$-parameterised inference module
enables the exploitation of the inter-task relationships to reason over each
local latent parameter. Thus, the introduced tokens can be refined with
relevant information from the heterogeneous context sets. By integrating meta-
knowledge and heterogeneous context sets, HNPs can reduce the negative
transfer of task-specific knowledge among heterogeneous tasks in each episode.
Please refer to Appendix E for algorithms.

##  3 Related Work

#### Multi-Task Learning.

Multi-task learning can operate in various settings [9]. Here we roughly
separate the settings of MTL into two branches: (1) Single-input multi-output
(SIMO) [24, 25, 26, 27, 28, 29, 30], where tasks are defined by different
supervision information for the same input. (2) Multi-input multi-output
(MIMO) [11, 10, 31, 32, 33, 34], where heterogeneous tasks follow different
data distributions. This work considers the MIMO setup of multi-task learning
with episodic training.

In terms of modeling methods, from a processing perspective, existing MTL
methods can be roughly categorized into two groups: (1) Probabilistic MTL
methods [11, 19, 35, 36, 37, 38, 39, 40, 41], which employ the Bayes framework
to characterize probabilistic dependencies among tasks. (2) Deep MTL models
[10, 32, 24, 25, 26, 42, 43, 44, 45, 46, 47, 48], which directly utilize deep
neural networks to discover information-sharing mechanisms across tasks.
However, deep MTL models rely on large amounts of training data and tend to
overfit when encountering the data-insufficiency problem. Meanwhile, previous
probabilistic MTL methods consider a small number of tasks that occur at the
same time, limiting their applicability in real-world systems.

#### Meta-Learning.

Meta-learning aims to find strategies to quickly adapt to unseen tasks with a
few examples [49, 4, 5, 50]. There exist a couple of branches in meta-learning
methods, such as metrics-based methods [6, 51, 52, 53, 54, 55, 56, 57] and
optimization-based methods [5, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68].
Our paper focuses on a probabilistic meta-learning method, namely neural
processes, that can quantify predictive uncertainty. Models in this family [7,
12, 13, 15, 16, 18, 69, 70, 71, 72, 73] can approximate stochastic processes
in neural networks. Vanilla NPs [12] usually encounter the expressiveness
bottleneck because their functional priors are not rich enough to generate
complicated functions [15, 16]. [7] introduces deterministic variables to
model predictive distributions for meta-learning scenarios directly. Most NP-
based methods only focus on a single task during inference [7, 12, 15, 16,
14], which leaves task-relatedness between heterogeneous tasks in a single
episode an open problem.

This paper combines multi-task learning and meta-learning paradigms to tackle
the data-insufficiency problem. Our work shares the high-level goal of
exploiting task-relatedness in an episode with [19, 74, 75]. Concerning the
multi-task scenarios, the main differences are: [19, 74, 75] handles multiple
attributes and multi-sensor data under the SIMO setting, while our work
performs for the MIMO setting where tasks are heterogeneous and distribution
shifts exist. Moreover, [76] theoretically addresses the conclusion that MTL
methods are powerful and efficient alternatives to gradient-based meta-
learning algorithms. However, our method inherits the advantages of multi-task
learning and meta-learning: simultaneously generalizing meta-knowledge from
past to new episodes and exploiting task-relatedness across heterogeneous
tasks in every single episode. Thus, our method is more suitable for solving
the data-insufficiency problem. Intuitive comparisons with related paradigms
such as cross-domain few-shot learning [77, 78, 79, 80, 81, 82], multimodal
meta-learning [83, 84, 85, 86, 87, 56] and cross-modality few-shot learning
[88, 89, 90] are provided in Appendix A.

##  4 Experiments

We evaluate the proposed HNPs and baselines on three benchmark datasets under
the episodic multi-task setup. Sec. 4.1 and Sec. 4.2 provide experimental
results for regression and classification, respectively. Ablation studies are
in Sec. 4.3. More comparisons with recent works on extra datasets are provided
in Appendix F. Additional results under the convectional MIMO setup without
episodic training can be found in Appendix G & H.

###  4.1 Episodic Multi-Task Regression

#### Dataset and Settings.

To evaluate the benefit of HNPs over typical NP baselines in uncertainty
quantification, we conduct experiments in several 1D regression tasks. The
baselines include conditional neural processes (CNPs [13]), vanilla neural
processes (NPs [12]), and attentive neural processes (ANPs [15]). As a toy
example, we construct multiple tasks with different task distributions: each
task’s input set is defined on separate intervals without overlap.

Figure 4: Performance comparisons on the episodic multi-task $1$-D function
regression using $5$ context points (black dots) for each task. Black curves
are ground truth, and blue ones are predicted results. The shadow regions are
$\pm 3$ standard derivations from the mean [18].

Given four different tasks in an episode, their input sets are
$\mathbf{x}^{1:4}_{\tau}$. Each input set contains a few instances, drawn
uniformly at random from separate intervals, such as
$\mathbf{x}^{1}_{\tau}\in[-4,-2)$, $\mathbf{x}^{2}_{\tau}\in[-2,0)$,
$\mathbf{x}^{3}_{\tau}\in[0,2)$, and $\mathbf{x}^{4}_{\tau}\in[2,4)$. All
tasks in an episode are related by sharing the same ground truth function.
Following [12, 18], function-fitting tasks are generated with Gaussian
processes (GPs). Here a zero mean Gaussian process
$y^{(0)}\sim\mathcal{GP}(0,k(\cdot,\cdot))$ is used to produce
$\mathbf{y}^{1:4}_{\tau}$ for the inputs from all tasks
$\mathbf{x}^{1:4}_{\tau}$. A radial basis kernel
$k(x,x{{}^{\prime}})=\sigma^{2}\exp(-(x-x{{}^{\prime}})^{2})/2l^{2})$, with
$l=0.4$ and $\sigma=1.0$ is used.

#### Results and Discussions.

As shown in Figure 4, CNPs, ANPs, and our HNPs exhibit more reasonable
uncertainty than NPs in Figure 4: lower variances are predicted around
observed (context) points with higher variances around unobserved points.
Furthermore, NPs and ANPs detrimentally impact the smoothness of the predicted
curves, whereas HNPs yield smoother predictive curves with reliable
uncertainty estimation. These observations suggest that integrating
correlation information across related tasks and meta-knowledge in HNPs can
improve uncertainty quantification in multi-task regression.

Table 1: Average negative log-likelihoods over target points from all tasks.

Methods CNPs NPs ANPs HNPs Avg. NLL 0.0935 0.8649 -0.1165 -0.5207

To quantify uncertainty we use the average negative log-likelihood (the lower,
the better). As shown in Table 1, our HNPs achieve a lower average negative
log-likelihood than baselines, demonstrating our method’s effectiveness in
uncertainty estimation.

###  4.2 Episodic Multi-task Classification

#### Datasets and Settings.

We use Office-Home [91] and DomainNet [92] as episodic multi-task
classification datasets. Office-Home contains images from four domains:
Artistic (A), Clipart (C), Product (P) and Real-world (R). Each domain
contains images from $65$ categories collected from office and home
environments. Note that all domains share the whole target space. The numbers
of meta-training classes and meta-test classes are $40$ and $25$,
respectively. There are about $15,500$ images in total. DomainNet has six
distinct domains: Clipart, Infograph, Painting, Quickdraw, Real and Sketch. It
includes approximately 0.6 million images distributed over $345$ categories.
The numbers of meta-training classes and meta-test classes are $276$ and $69$,
respectively. Here one domain corresponds to a specific task distribution in
the episodic multi-task setting.

When it comes to the episodic multi-task classification, we compare HNPs with
the following three branches: (1) Multi-task learning methods: ERM [93]
directly expands the training set of the current task with samples of related
tasks. VMTL [11] is one of the state-of-the-art under the MIMO setting of
multi-task learning. (2) Meta-learning methods: MAML [5], Proto.Net [6] and
DGPs [94] address each task separately with no mechanism to leverage task-
relatedness in a single episode. (3) Methods from the NP family: CNPs [13] and
NPs [12] are established methods in the NP family. TNP-D [95] is recent NP
work in sequential decision-making for a single task in each episode.

#### Results and Discussions.

The experimental results for episodic multi-task classification on Office-Home
and DomainNet are reported in Table 2. We use the average accuracy across all
task distributions as the evaluation metric. It can be seen that HNPs
consistently outperform all baseline methods, demonstrating the effectiveness
of HNPs in handling each task with limited data under the episodic multi-task
classification setup.

NPs and CNPs do not work well under all episodic multi-task classification
cases. This can be attributed to their limited expressiveness of the global
representation and the weak capability to extract discriminative information
from multiple contexts. In contrast, HNPs explicitly abstract discriminative
information for each task in the episode with the help of local latent
parameters, enhancing the expressiveness of the functional prior.

Table 2: Comparative results (95% confidence interval) for episodic multi-task
classification on Office-Home and DomainNet. Best results are indicated in
bold.

Office-Home DomainNet 4-task 5-way 4-task 20-way 6-task 5-way 6-task 20-way
Method 1-shot 5-shot 1-shot 5-shot 1-shot 5-shot 1-shot 5-shot ERM [93] 66.04
$\pm$0.61 73.62 $\pm$0.55 39.25 $\pm$0.24 47.14 $\pm$0.18 59.95 $\pm$0.52
68.52 $\pm$0.44 38.62 $\pm$0.22 47.85 $\pm$0.20 VMTL [11] 49.71 $\pm$0.48
65.75 $\pm$0.47 27.50 $\pm$0.14 42.82 $\pm$0.13 42.24 $\pm$0.39 57.37
$\pm$0.43 18.05 $\pm$0.11 31.38 $\pm$0.15 MAML [5] 60.58 $\pm$0.60 75.29
$\pm$0.53 34.29 $\pm$0.19 48.39 $\pm$0.20 53.21 $\pm$0.46 65.24 $\pm$0.47
17.10 $\pm$0.12 20.35 $\pm$0.14 Proto. Net. [6] 57.19 $\pm$0.53 74.97
$\pm$0.46 32.72 $\pm$0.18 49.75 $\pm$0.16 53.71 $\pm$0.48 68.80 $\pm$0.42
31.90 $\pm$0.19 47.59 $\pm$0.18 DGPs [94] 65.89 $\pm$0.53 79.96 $\pm$0.38
31.48 $\pm$0.18 49.46 $\pm$0.18 50.93 $\pm$0.42 63.32 $\pm$0.38 25.46
$\pm$0.15 38.63 $\pm$0.17 CNPs [13] 43.33 $\pm$0.56 55.07 $\pm$0.63 10.57
$\pm$0.10 12.02 $\pm$0.11 37.90 $\pm$0.45 40.53 $\pm$0.44 5.12 $\pm$0.10 5.14
$\pm$0.10 NPs [12] 33.66 $\pm$0.48 53.99 $\pm$0.60 5.25 $\pm$0.16 11.40
$\pm$0.11 20.58 $\pm$0.51 20.53 $\pm$0.53 5.12 $\pm$0.09 5.11 $\pm$0.09 TNP-D
[95] 65.49 $\pm$0.53 78.94 $\pm$0.43 41.61 $\pm$0.22 59.19 $\pm$0.21 49.10
$\pm$0.42 67.39 $\pm$0.40 28.83 $\pm$0.17 47.69 $\pm$0.18 HNPs 76.29 $\pm$0.51
80.80 $\pm$0.42 51.82 $\pm$0.23 59.97 $\pm$0.18 62.36 $\pm$0.53 69.38
$\pm$0.42 39.32 $\pm$0.23 48.56 $\pm$0.19

We also find that HNPs significantly surpass other baselines on 1-shot Office-
Home and DomainNet, both under the 4/6-task 5-way and 4/6-task 20-way
settings. This further implies that HNPs can circumvent the effect of the
problem of data-insufficiency by simultaneously exploiting task-relatedness
across heterogeneous tasks and meta-knowledge among episodes.

###  4.3 Ablation Studies

#### Influence of Hierarchical Latent Variables.

We first investigate the roles of the global latent representation
$\mathbf{z}^{m}_{\tau}$ and the local latent parameters
$\mathbf{w}^{m}_{\tau,1:O}$ by leaving out individual inference modules. These
experiments are performed on Office-home under the 4-task 5-way 1-shot
setting. We report the detailed performance for tasks sampled from a single
task distribution (A/C/P/R) and the average accuracy across all task
distributions (Avg.) in Table 3. The variants without specific latent
variables are included in the comparison by removing the corresponding
inference modules.

Table 3: Effectiveness of global latent representations
$\mathbf{z}_{\tau}^{m}$ and local latent parameters
$\mathbf{w}_{\tau,1:O}^{m}$ in the model. ✓ and ✗ denote whether the variants
of HNPs have the corresponding latent variable or not.

$\mathbf{z}_{\tau}^{m}$ $\mathbf{w}_{\tau,1:O}^{m}$ A C P R Avg. ✗ ✗ 62.64
$\pm$0.72 56.87 $\pm$0.71 75.18 $\pm$0.79 73.68 $\pm$0.77 67.09 $\pm$0.63 ✗ ✓
69.39 $\pm$0.60 63.10 $\pm$0.61 80.66 $\pm$0.67 79.99 $\pm$0.62 73.29
$\pm$0.51 ✓ ✗ 67.02 $\pm$0.67 60.70 $\pm$0.69 78.26 $\pm$0.76 78.47 $\pm$0.72
71.11 $\pm$0.59 ✓ ✓ 73.31 $\pm$0.63 64.92 $\pm$0.68 83.38 $\pm$0.66 83.54
$\pm$0.64 76.29 $\pm$0.51

As shown in Table 3, both $\mathbf{z}^{m}_{\tau}$ and
$\mathbf{w}^{m}_{\tau,1:O}$ benefit overall performance. Our method with
hierarchical latent variables performs $9.20\%$ better than the variant
without both latent variables, $3.00\%$ better than the variant without
$\mathbf{z}^{m}_{\tau}$, and $5.18\%$ better than the variant without
$\mathbf{w}^{m}_{\tau,1:O}$. This indicates that latent variables of HNPs
complement each other in representing context sets from multiple tasks and
meta-knowledge. The variant without $\mathbf{w}^{m}_{\tau,1:O}$ underperforms
the variant without $\mathbf{z}^{m}_{\tau}$ by $2.18\%$, in terms of the
average accuracy. This demonstrates that $\mathbf{z}^{m}_{\tau}$ suffers more
from the expressiveness bottleneck than $\mathbf{w}^{m}_{\tau,1:O}$, weakening
the models’ discriminative ability. For classification, local latent
parameters are more crucial than a global latent representation in revealing
the discriminating knowledge from multiple heterogeneous context sets.

#### Influence of Transformer-Structured Inference Modules.

To further understand our transformer-structured inference modules (Trans. w
learnable tokens), we examine the performance against two other options:
inference modules that solely utilize a multi-layer perceptron (MLP) and the
variants that do not incorporate any learnable tokens (Trans. w/o learnable
tokens). We also compare the probabilistic and deterministic versions of such
inference modules. The deterministic variants consider the deterministic
embedding for the hierarchical latent variables.

Table 4: Performance comparisons between our transformer inference modules
(Trans. w learnable tokens) and other alternatives.

Inference networks 1-shot 5-shot Deterministic MLP 64.93 $\pm$0.66 72.39
$\pm$0.56 Trans. w/o learnable tokens 70.22 $\pm$0.62 76.15 $\pm$0.54 Trans. w
learnable tokens 70.61 $\pm$0.56 76.70 $\pm$0.50 Probabilistic MLP 73.30
$\pm$0.59 77.94 $\pm$0.48 Trans. w/o learnable tokens 75.25 $\pm$0.55 80.42
$\pm$0.47 Trans. w learnable tokens 76.29 $\pm$0.51 80.80 $\pm$0.42

As shown in Table 4, our inference modules consistently outperform the
variants, regardless of whether the inference network is probabilistic or
deterministic. When using the probabilistic one, our inference modules
respectively achieve $1.04\%$ and $2.99\%$ performance gains over Trans. w/o
learnable tokens and MLP under the 4-task 5-way 1-shot setting. This implies
the importance of learnable tokens and task-relatedness in formulating
transformer-structured inference modules, which reduces negative transfer
among heterogeneous tasks in each meta-test episode. Moreover, the variants
with probabilistic inference modules consistently beat deterministic ones in
performance, demonstrating the advantages of considering uncertainty during
modeling and inference.

Table 5: Performance comparisons of different implementations of generating
each local latent parameter $\mathbf{w}^{m}_{\tau,o}$ from the condition
$\mathbf{z}^{m}_{\tau}$ and $\mathcal{C}_{\tau}^{1:M}$.

Methods A C P R Avg. Concat 65.69 $\pm$0.59 58.64 $\pm$0.61 77.54 $\pm$0.68
77.10 $\pm$0.64 69.74 $\pm$0.51 Add 69.92 $\pm$0.69 63.73 $\pm$0.71 78.81
$\pm$0.77 79.03 $\pm$0.78 72.87 $\pm$0.61 Ours 73.31 $\pm$0.63 64.92 $\pm$0.68
83.38 $\pm$0.66 83.54 $\pm$0.64 76.29 $\pm$0.51

#### Effects of Different Ways to Generate Local Latent Parameters.

We investigate the effects of different ways to generate each
$\mathbf{w}^{m}_{\tau,o}$ from the shared condition $\mathbf{z}^{m}_{\tau}$
and $\mathcal{C}_{\tau}^{1:M}$. Given a Monte Carlo sample of global latent
variables as ${\mathbf{z}^{m}_{\tau}}^{(i)}$, in Table 5, we compare with two
alternatives: 1) Concat directly concatenates each context feature and
${\mathbf{z}^{m}_{\tau}}^{(i)}$, and takes the concatenation as inputs of the
transformer-structured inference network $\phi$. 2) Add sums up each context
feature and ${\mathbf{z}^{m}_{\tau}}^{(i)}$ and takes the result as the input.
3) Ours incorporates ${\mathbf{z}^{m}_{\tau}}^{(i)}$ into the transformer-
structured inference module by merging it with the refined learnable tokens in
Eq. (10). As shown in Table 5, Ours consistently performs the best. This
implies that incorporating the conditional variables into the inference module
is more effective than the direct combinations of
${\mathbf{z}^{m}_{\tau}}^{(i)}$ and instance features.

#### Effects of More "Shots" or "Classes".

To investigate the effects of more "shots" or "classes" in the episodic multi-
task classification setup, we conduct experiments by increasing $K$ or $O$ in
the defined $M$-task $O$-way $K$-shot setup.

Table 6: Performance comparisons on Office-Home under the 4-task 5-way K-shot
setup.

Methods 1-shot 5-shot 10-shot 20-shot TNP-D 65.49 $\pm$0.53 78.94 $\pm$0.43
80.81 $\pm$0.32 81.12 $\pm$0.68 HNPs 76.29 $\pm$0.51 80.80 $\pm$0.42 81.28
$\pm$0.38 81.56 $\pm$0.36

As shown in Table 6, the proposed HNPs have more advantages over the baseline
method with the context data points below ten shots. With shots larger than
ten, both methods will reach a performance bottleneck.

Moreover, Table 7 shows that our method consistently outperforms the baseline
method as the number of classes increases from 20 to 40 in step 5. However,
the performance gap between them narrows slightly with more classes. The main
reason could be that the setting with more classes suffers from less data
insufficiency.

Table 7: Performance comparisons on DomainNet under the 6-task O-way 1-shot
setup.

Methods 5-way 20-way 25-way 30-way 35-way 40-way TNP-D 49.10 $\pm$0.42 28.83
$\pm$0.17 25.93 $\pm$0.14 24.08 $\pm$0.12 22.62 $\pm$0.11 21.64 $\pm$0.53 HNPs
62.36 $\pm$0.53 39.32 $\pm$0.23 35.72 $\pm$0.19 32.27 $\pm$0.17 31.27
$\pm$0.14 29.31 $\pm$0.13

Figure 5: Average accuracy and runtime of HNPs with different numbers of Monte
Carlo samples. $N_{z}$ and $N_{w}$ are sampling numbers of
$\mathbf{z}^{m}_{\tau}$ and $\mathbf{w}^{m}_{\tau,1:O}$, respectively.

#### Sensitivity to the Number of Monte Carlo Samples.

For the hierarchical latent variables in the HNPs, we investigate the model’s
sensitivity to the number of Monte Carlo samples. Specifically, the sampling
number of the global latent representation $\mathbf{z}^{m}_{\tau}$ and local
latent parameters $\mathbf{w}^{m}_{\tau,1:O}$ varies from $1$ to $30$. We
examine on Office-Home under the 4-task 5-way 1-shot setting. In Figure 5, the
runtime per iteration grows rapidly as the number of samples increases.
However, there is no clear correlation between the performance and the number
of Monte Carlo samples. There are two sweet spots in terms of average
accuracy, one of which has favorable computation time. Hence, we set $N_{z}$
and $N_{w}$ to $5$ and $10$, respectively.

Table 8: Inference time of different NP-based methods.

Methods CNPs NPs TNP-D HNPs Inference time(s) 0.04 0.05 0.08 0.15

We also investigate the inference time of NP-based models per iteration on
Office-Home under the 4task5way1shot setup. As shown in Table 8, our model
needs more inference time than other NP-based methods for performance gains.
The cost mainly comes from inferring the designed hierarchical latent
variables; however, we consider this a worthwhile trade-off for the extra
performance.

##  5 Conclusion

#### Technical Discussion.

This work develops heterogeneous neural processes by introducing hierarchical
latent variables and transformer-structured inference modules for episodic
multi-task learning. With the help of heterogeneous context information and
meta-knowledge, the proposed model can exploit task-relatedness, reason about
predictive function distributions, and efficiently distill past knowledge to
unseen heterogeneous tasks with limited data.

#### Limitation & Extension.

Although the hierarchical probabilistic framework could mitigate the
expressiveness bottleneck, the model needs more inference time than other NP-
based methods for performance gains. Besides, the proposed method requires the
target space to be the same across all tasks in a single episode. This
requirement could limit the method’s applicability in realistic scenarios
where target spaces may differ across tasks. Our work could be extended to the
new case without the shared target spaces, where the model should construct
higher-order task-relatedness to improve knowledge sharing among tasks. Our
code 222https://github.com/autumn9999/HNPs.git is provided to facilitate such
extensions.

## Acknowledgment

This work is financially supported by the Inception Institute of Artificial
Intelligence, the University of Amsterdam and the allowance Top consortia for
Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic
Affairs and Climate Policy.

## References

  * Goodfellow et al. [2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  _Deep Learning_.  MIT Press, 2016.  http://www.deeplearningbook.org.
  * Xu et al. [2021] Shichao Xu, Lixu Wang, Yixuan Wang, and Qi Zhu.  Weak adaptation learning: Addressing cross-domain data insufficiency with weak annotator.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2021\.
  * Johnson and Khoshgoftaar [2019] Justin M Johnson and Taghi M Khoshgoftaar.  Survey on deep learning with class imbalance.  _Journal of Big Data_ , 6(1):1–54, 2019.
  * Vinyals et al. [2016] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  In _Advances in Neural Information Processing Systems_ , 2016.
  * Finn et al. [2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , 2017.
  * Snell et al. [2017] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Requeima et al. [2019] James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner.  Fast and flexible multi-task classification using conditional neural adaptive processes.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Caruana [1997] Rich Caruana.  Multitask learning.  _Machine learning_ , 28(1):41–75, 1997.
  * Zhang and Yang [2021] Yu Zhang and Qiang Yang.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * Long et al. [2017] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and S Yu Philip.  Learning multiple tasks with multilinear relationship networks.  In _Advances in neural information processing systems_ , 2017.
  * Shen et al. [2021] Jiayi Shen, Xiantong Zhen, Marcel Worring, and Ling Shao.  Variational multi-task learning with gumbel-softmax priors.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Garnelo et al. [2018a] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh.  Neural processes.  _arXiv preprint arXiv:1807.01622_ , 2018a.
  * Garnelo et al. [2018b] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami.  Conditional neural processes.  In _International Conference on Machine Learning_ , 2018b.
  * Bruinsma et al. [2023] Wessel Bruinsma, Stratis Markou, James Requeima, Andrew Y. K. Foong, Tom Andersson, Anna Vaughan, Anthony Buonomo, Scott Hosking, and Richard E Turner.  Autoregressive conditional neural processes.  In _International Conference on Learning Representations_ , 2023.
  * Kim et al. [2019] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh.  Attentive neural processes.  In _International Conference on Learning Representations_ , 2019.
  * Wang and van Hoof [2022] Qi Wang and Herke van Hoof.  Learning expressive meta-representations with mixture of expert neural processes.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Ravi and Larochelle [2017] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In _International Conference on Learning Representations_ , 2017.
  * Wang and Van Hoof [2020] Qi Wang and Herke Van Hoof.  Doubly stochastic variational inference for neural processes with hierarchical latent variables.  In _International Conference on Machine Learning_ , 2020.
  * Kim et al. [2021a] Donggyun Kim, Seongwoong Cho, Wonkwang Lee, and Seunghoon Hong.  Multi-task processes.  _arXiv preprint arXiv:2110.14953_ , 2021a.
  * Guo et al. [2023] Zongyu Guo, Cuiling Lan, Zhizheng Zhang, Yan Lu, and Zhibo Chen.  Versatile neural processes for learning implicit neural representations.  In _International Conference on Learning Representations_ , 2023.
  * Klenke [2013] Achim Klenke.  _Probability theory: a comprehensive course_.  Springer Science & Business Media, 2013.
  * Kim et al. [2021b] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , 2021b.
  * Sønderby et al. [2016] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther.  Ladder variational autoencoders.  In _Advances in Neural Information Processing Systems_ , 2016.
  * Kendall et al. [2018] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018\.
  * Liu et al. [2019] Shikun Liu, Edward Johns, and Andrew J Davison.  End-to-end multi-task learning with attention.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019\.
  * Misra et al. [2016] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert.  Cross-stitch networks for multi-task learning.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016\.
  * Liu et al. [2020] Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang.  Towards impartial multi-task learning.  In _International Conference on Learning Representations_ , 2020.
  * Sener and Koltun [2018] Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Zamir et al. [2018] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018.
  * Bhattacharjee et al. [2022] Deblina Bhattacharjee, Tong Zhang, Sabine Süsstrunk, and Mathieu Salzmann.  Mult: an end-to-end multitask learning transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Zhang et al. [2021a] Yi Zhang, Yu Zhang, and Wei Wang.  Multi-task learning via generalized tensor trace norm.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 2021a.
  * Shen et al. [2022] Jiayi Shen, Zehao Xiao, Xiantong Zhen, Cees GM Snoek, and Marcel Worring.  Association graph learning for multi-task classification with category shifts.  _arXiv preprint arXiv:2210.04637_ , 2022.
  * Liang et al. [2022] Yunlong Liang, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou.  Scheduled multi-task learning for neural chat translation.  _arXiv preprint arXiv:2205.03766_ , 2022.
  * Zhang et al. [2022a] Yi Zhang, Yu Zhang, and Wei Wang.  Learning linear and nonlinear low-rank structure in multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 2022a.
  * Bakker and Heskes [2003] Bart Bakker and Tom Heskes.  Task clustering and gating for bayesian multitask learning.  _Journal of Machine Learning Research_ , 2003.
  * Yu et al. [2005] Kai Yu, Volker Tresp, and Anton Schwaighofer.  Learning gaussian processes from multiple tasks.  In _International Conference on Machine Learning_ , 2005.
  * Titsias and Lázaro-Gredilla [2011] Michalis K Titsias and Miguel Lázaro-Gredilla.  Spike and slab variational inference for multi-task and multiple kernel learning.  In _Advances in neural information processing systems_ , 2011.
  * Lawrence and Platt [2004] Neil D Lawrence and John C Platt.  Learning to learn with the informative vector machine.  In _International Conference on Machine Learning_ , 2004.
  * Yousefi et al. [2019] Fariba Yousefi, Michael Thomas Smith, and Mauricio A Álvarez.  Multi-task learning for aggregated data using gaussian processes.  _arXiv preprint arXiv:1906.09412_ , 2019.
  * Oyen and Lane [2012] Diane Oyen and Terran Lane.  Leveraging domain knowledge in multitask bayesian network structure learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2012.
  * Qian et al. [2020] Weizhu Qian, Bowei Chen, Yichao Zhang, Guanghui Wen, and Franck Gechter.  Multi-task variational information bottleneck.  _arXiv preprint arXiv:2007.00339_ , 2020.
  * Strezoski et al. [2019a] Gjorgji Strezoski, Nanne van Noord, and Marcel Worring.  Learning task relatedness in multi-task learning for images in context.  In _International Conference on Multimedia Retrieval_ , 2019a.
  * Sun et al. [2019] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko.  Adashare: Learning what to share for efficient deep multi-task learning.  _arXiv preprint arXiv:1911.12423_ , 2019.
  * Strezoski et al. [2019b] Gjorgji Strezoski, Nanne van Noord, and Marcel Worring.  Many task learning with task routing.  In _IEEE International Conference on Computer Vision_ , 2019b.
  * Yu et al. [2020] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.  Gradient surgery for multi-task learning.  _arXiv preprint arXiv:2001.06782_ , 2020.
  * Liu et al. [2021] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu.  Conflict-averse gradient descent for multi-task learning.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Fifty et al. [2021] Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn.  Efficiently identifying task groupings for multi-task learning.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Guo et al. [2020a] Pengsheng Guo, Chen-Yu Lee, and Daniel Ulbricht.  Learning to branch for multi-task learning.  In _International Conference on Machine Learning_ , 2020a.
  * Thrun and Pratt [1998] Sebastian Thrun and Lorien Pratt.  Learning to learn: Introduction and overview.  In _Learning to learn_ , pages 3–17. Springer, 1998.
  * Hospedales et al. [2021] Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J Storkey.  Meta-learning in neural networks: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Allen et al. [2019] Kelsey R Allen, Evan Shelhamer, Hanul Shin, and Joshua B Tenenbaum.  Infinite mixture prototypes for few-shot learning.  In _International Conference on Machine Learning_ , 2019.
  * Oreshkin et al. [2018] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste.  Tadam: Task dependent adaptive metric for improved few-shot learning.  In _Advances in neural information processing systems_ , 2018.
  * Yoon et al. [2019] Sung Whan Yoon, Jun Seo, and Jaekyun Moon.  Tapnet: Neural network augmented with task-adaptive projection for few-shot learning.  In _International Conference on Machine Learning_ , 2019.
  * Garcia and Bruna [2018] Victor Garcia and Joan Bruna.  Few-shot learning with graph neural networks.  In _International Conference on Learning Representations_ , 2018.
  * Cao et al. [2019] Tianshi Cao, Marc Law, and Sanja Fidler.  A theoretical analysis of the number of shots in few-shot learning.  _arXiv preprint arXiv:1909.11722_ , 2019.
  * Triantafillou et al. [2019] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle.  Meta-dataset: A dataset of datasets for learning to learn from few examples.  _arXiv preprint arXiv:1903.03096_ , 2019.
  * Sung et al. [2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018\.
  * Nichol et al. [2018] Alex Nichol, Joshua Achiam, and John Schulman.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_ , 2018.
  * Rusu et al. [2019] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell.  Meta-learning with latent embedding optimization.  In _International Conference on Learning Representations_ , 2019.
  * Sung et al. [2017] Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang.  Learning to learn: Meta-critic networks for sample efficient learning.  _arXiv preprint arXiv:1706.09529_ , 2017.
  * Li et al. [2017] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li.  Meta-sgd: Learning to learn quickly for few-shot learning.  _arXiv preprint arXiv:1707.09835_ , 2017.
  * Gordon et al. [2019] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner.  Meta-learning probabilistic inference for prediction.  In _International Conference on Learning Representations_ , 2019.
  * Edwards and Storkey [2016] Harrison Edwards and Amos Storkey.  Towards a neural statistician.  _arXiv preprint arXiv:1606.02185_ , 2016.
  * Finn et al. [2018] Chelsea Finn, Kelvin Xu, and Sergey Levine.  Probabilistic model-agnostic meta-learning.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Sæmundsson et al. [2018] Steindór Sæmundsson, Katja Hofmann, and Marc Peter Deisenroth.  Meta reinforcement learning with latent variable gaussian processes.  _arXiv preprint arXiv:1803.07551_ , 2018.
  * Baik et al. [2021] Sungyong Baik, Junghoon Oh, Seokil Hong, and Kyoung Mu Lee.  Learning to forget for meta-learning via task-and-layer-wise attenuation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Choi et al. [2021] Myungsub Choi, Janghoon Choi, Sungyong Baik, Tae Hyun Kim, and Kyoung Mu Lee.  Test-time adaptation for video frame interpolation via meta-learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Raghu et al. [2019] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals.  Rapid learning or feature reuse? towards understanding the effectiveness of maml.  _arXiv preprint arXiv:1909.09157_ , 2019.
  * Wei et al. [2021] Ying Wei, Peilin Zhao, and Junzhou Huang.  Meta-learning hyperparameter performance prediction with neural processes.  In _International Conference on Machine Learning_ , 2021.
  * Markou et al. [2022] Stratis Markou, James Requeima, Wessel P Bruinsma, Anna Vaughan, and Richard E Turner.  Practical conditional neural processes via tractable dependent predictions.  _arXiv preprint arXiv:2203.08775_ , 2022.
  * Ye and Yao [2022] Zesheng Ye and Lina Yao.  Contrastive conditional neural processes.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2022\.
  * Kim et al. [2022] Mingyu Kim, Kyeongryeol Go, and Se-Young Yun.  Neural processes with stochastic attention: Paying more attention to the context dataset.  _arXiv preprint arXiv:2204.05449_ , 2022.
  * Wang et al. [2023] Qi Wang, Marco Federici, and Herke van Hoof.  Bridge the inference gaps of neural processes via expectation maximization.  In _International Conference on Learning Representations_ , 2023.
  * Song et al. [2022] Xiaozhuang Song, Shun Zheng, Wei Cao, James Yu, and Jiang Bian.  Efficient and effective multi-task grouping via meta learning on task combinations.  _Advances in Neural Information Processing Systems_ , 2022.
  * Upadhyay et al. [2023] Richa Upadhyay, Prakash Chandra Chhipa, Ronald Phlypo, Rajkumar Saini, and Marcus Liwicki.  Multi-task meta learning: learn how to adapt to unseen tasks.  In _2023 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2023.
  * Wang et al. [2021a] Haoxiang Wang, Han Zhao, and Bo Li.  Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation.  In _International Conference on Machine Learning_ , 2021a.
  * Tseng et al. [2020] Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang.  Cross-domain few-shot classification via learned feature-wise transformation.  In _International Conference on Learning Representations_ , 2020.
  * Chen et al. [2019] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  _arXiv preprint arXiv:1904.04232_ , 2019.
  * Guo et al. [2020b] Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris.  A broader study of cross-domain few-shot learning.  In _European Conference on Computer Vision_ , 2020b.
  * Du et al. [2021] Yingjun Du, Xiantong Zhen, Ling Shao, and Cees G M Snoek.  MetaNorm: Learning to normalize few-shot batches across domains.  In _International Conference on Learning Representations_ , 2021.
  * Das et al. [2022] Debasmit Das, Sungrack Yun, and Fatih Porikli.  Confess: A framework for single source cross-domain few-shot learning.  In _International Conference on Learning Representations_ , 2022.
  * Li et al. [2022] Wei-Hong Li, Xialei Liu, and Hakan Bilen.  Cross-domain few-shot learning with task-specific adapters.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7161–7170, 2022.
  * Vuorio et al. [2019] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim.  Multimodal model-agnostic meta-learning via task-aware modulation.  In _Advances in neural information processing systems_ , 2019.
  * Vuorio et al. [2018] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim.  Toward multimodal model-agnostic meta-learning.  In _arXiv preprint arXiv:1812.07172_ , 2018.
  * Yao et al. [2019] Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li.  Hierarchically structured meta-learning.  In _International Conference on Machine Learning_ , 2019.
  * Abdollahzadeh et al. [2021] Milad Abdollahzadeh, Touba Malekzadeh, and Ngai-Man Man Cheung.  Revisit multimodal meta-learning through the lens of multi-task learning.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Chen and Zhang [2021] Jiayi Chen and Aidong Zhang.  Hetmaml: Task-heterogeneous model-agnostic meta-learning for few-shot learning across modalities.  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021.
  * Xing et al. [2019] Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O Pinheiro.  Adaptive cross-modal few-shot learning.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Pahde et al. [2018] Frederik Pahde, Patrick Jähnichen, Tassilo Klein, and Moin Nabi.  Cross-modal hallucination for few-shot fine-grained recognition.  _arXiv preprint arXiv:1806.05147_ , 2018.
  * Pahde et al. [2021] Frederik Pahde, Mihai Puscas, Tassilo Klein, and Moin Nabi.  Multimodal prototypical networks for few-shot learning.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021.
  * Venkateswara et al. [2017] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017\.
  * Peng et al. [2019] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In _IEEE International Conference on Computer Vision_ , 2019.
  * Gulrajani and Lopez-Paz [2020] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  _arXiv preprint arXiv:2007.01434_ , 2020.
  * Wang et al. [2021b] Ze Wang, Zichen Miao, Xiantong Zhen, and Qiang Qiu.  Learning to learn dense gaussian processes for few-shot learning.  In _Advances in Neural Information Processing Systems_ , 2021b.
  * Nguyen and Grover [2022] Tung Nguyen and Aditya Grover.  Transformer neural processes: Uncertainty-aware meta learning via sequence modeling.  _arXiv preprint arXiv:2207.04179_ , 2022.
  * Upadhyay et al. [2022] Richa Upadhyay, Prakash Chandra Chhipa, Ronald Phlypo, Rajkumar Saini, and Marcus Liwicki.  Multi-task meta learning: learn how to adapt to unseen tasks.  _arXiv preprint arXiv:2210.06989_ , 2022.
  * Zhang et al. [2022b] Qianqian Zhang, Xinru Liao, Quan Liu, Jian Xu, and Bo Zheng.  Leaving no one behind: A multi-scenario multi-task meta learning approach for advertiser modeling.  In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ , 2022b.
  * Cao et al. [2021] Kaidi Cao, Jiaxuan You, and Jure Leskovec.  Relational multi-task learning: Modeling relations between data and tasks.  In _International Conference on Learning Representations_ , 2021.
  * Zhang et al. [2021b] Huaiwen Zhang, Shengsheng Qian, Quan Fang, and Changsheng Xu.  Multi-modal meta multi-task learning for social media rumor detection.  _IEEE Transactions on Multimedia_ , 24:1449–1459, 2021b.
  * Han et al. [2022] Chu Han, Huasheng Yao, Bingchao Zhao, Zhenhui Li, Zhenwei Shi, Lei Wu, Xin Chen, Jinrong Qu, Ke Zhao, Rushi Lan, et al.  Meta multi-task nuclei segmentation with fewer training samples.  _Medical Image Analysis_ , 2022.
  * Chen and Yeh [2021] Guan-Yuan Chen and Ya-Fen Yeh.  Mmtl: The meta multi-task learning for aspect category sentiment analysis.  In _Proceedings of the 33rd Conference on Computational Linguistics and Speech Processing_ , 2021.
  * Achille et al. [2019] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona.  Task2vec: Task embedding for meta-learning.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019.
  * Nguyen et al. [2021] Cuong C Nguyen, Thanh-Toan Do, and Gustavo Carneiro.  Probabilistic task modelling for meta-learning.  In _Uncertainty in Artificial Intelligence_ , 2021.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kingma et al. [2015] Durk P Kingma, Tim Salimans, and Max Welling.  Variational dropout and the local reparameterization trick.  In _Advances in neural information processing systems_ , 2015.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Yao et al. [2021] Huaxiu Yao, Linjun Zhang, and Chelsea Finn.  Meta-learning with fewer tasks through task interpolation.  _arXiv preprint arXiv:2106.02695_ , 2021.
  * Saenko et al. [2010] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In _European Conference on Computer Vision_ , 2010.
  * Kulis et al. [2011] Brian Kulis, Kate Saenko, and Trevor Darrell.  What you saw is not what you get: Domain adaptation using asymmetric kernel transforms.  In _CVPR 2011_. IEEE, 2011.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Gong et al. [2012] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman.  Geodesic flow kernel for unsupervised domain adaptation.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2012\.
  * Griffin et al. [2007] Gregory Griffin, Alex Holub, and Pietro Perona.  Caltech-256 object category dataset.  _Dataset Report_ , 2007.
  * Buda et al. [2019] Mateusz Buda, Ashirbani Saha, and Maciej A Mazurowski.  Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm.  _Computers in biology and medicine_ , 109:218–225, 2019\.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2015.

###### Contents

  1. 1 Introduction
  2. 2 Methodology
    1. 2.1 Modeling and Inference of Heterogeneous Neural Processes
    2. 2.2 Transformer-Structured Inference Module
  3. 3 Related Work
  4. 4 Experiments
    1. 4.1 Episodic Multi-Task Regression
    2. 4.2 Episodic Multi-task Classification
    3. 4.3 Ablation Studies
  5. 5 Conclusion
  6. A Frequently Asked Questions
  7. B Properties of Valid Exchangeable Stochastic Processes
    1. B.1 Proof of Exchangeability Consistency
    2. B.2 Proof of Marginalization Consistency
  8. C Tractable and Scalable Optimization
    1. C.1 Derivation of Approximate ELBO for HNPs
    2. C.2 Meta-Training Objective
    3. C.3 Meta-Test Prediction
  9. D More Experimental Details
    1. D.1 Transformer-structured Inference Modules in Regression Scenarios
    2. D.2 Backbone and Training Details
  10. E Algorithm of the proposed HNPs
  11. F More Experimental Results under Episodic Multi-Task Setup
    1. F.1 Effects of More “Tasks”
    2. F.2 4task20way1shot v.s. 20way4shot
    3. F.3 Comparisons with More Recent Works
    4. F.4 Comparisons on Another Benchmark Dataset
  12. G More Experimental Results under Conventional Multi-Task Setup
    1. G.1 Conventional Multi-Task Regression
    2. G.2 Conventional Multi-Task Classification
  13. H Application to Brain Image Segmentation

##  Appendix A Frequently Asked Questions

In this section, we list frequently asked questions from researchers who help
proofread this manuscript. These raised questions might also be relevant for
others and help in better understanding the paper, so we include more detailed
discussions here.

#### Connections between different settings.

This work considers the multi-input multi-output setting of multi-task
learning under the episodic training mechanism.

As shown in Table 9, we use "Heterogeneous tasks" to distinguish the different
branches of multi-task learning: (1) single-input multi-output (SIMO)
considers different tasks which have the same input and different supervision
information. (2) multi-input multi-output (MIMO) considers heterogeneous
tasks, which have different inputs and follow different data distributions.
All tasks are related since they share the target space. This setting
encourages deep models to deal with the insufficient data of each task by
aggregating the training data from related tasks in the spirit of data
augmentation.

Meanwhile, "Episodic training" is used to describe the data-feeding strategy.
Multi-task meta-learning also benefits from episodic training, but it follows
the SIMO setting in every single episode and cannot sufficiently handle
heterogeneous tasks. In our work, episodic multi-task learning is designed
based on the MIMO setting, suffering from distribution shifts between
heterogeneous tasks. In addition, we note that conventional meta-learning
follows the "Episodic training" mechanism but focuses on single-task learning
in each episode. Thus, "Heterogeneous tasks" is not available here (-). More
details are left in Table (9).

Table 9: Connections between different settings. The symbols ✓ and ✗ indicate
whether or not the specific setting has the corresponding characteristic.

Settings Methods Episodic training Heterogeneous tasks single-input multi-
output (SIMO) [24, 25, 26, 27, 28, 29, 30] ✗ ✗ multi-input multi-output (MIMO)
[11, 10, 31, 32, 33, 34] ✗ ✓ conventional meta-learning [4, 5, 6, 7, 12, 13] ✓
- multi-task meta-learning [96, 97, 98, 19, 99, 100, 101] ✓ ✗ episodic multi-
task learning This paper ✓ ✓

#### Problem scope.

In episodic multi-task learning, we restrict the scope of the problem to the
case where tasks in the same episode are related and share the same target
space. There are two main reasons: (1) we follow the MIMO setting of multi-
task learning in every single episode, where the same target space assures the
existence of the knowledge shared among tasks. (2) As demonstrated in recent
works [102, 103], meta-learning tasks generated from the same categories or
taxonomic clusters are closer. This also implies that tasks with the same
target space are related.

#### Differences from other episodic single-task setups.

Based on episodic training, there are several approaches related to the
setting of episodic multi-task learning: (1) cross-domain few-shot learning
addresses few-shot learning under domain shifts [77]. Several models [78, 77,
79, 80, 81, 82] train a model on a single source domain or several source
domains and then generalize it to other domains. In contrast, our research
emphasizes the domain shifts within individual episodes rather than among
them. (2) multimodal meta-learning extends few-shot learning from a single
input-label domain to multiple different input-label domains [83]. These
methods [84, 83, 56, 85, 86, 87] design a meta-learner that can handle tasks
from distinct distributions in sequence. Our work centers on simultaneously
dealing with several related tasks within a meta-training or meta-test
episode. (3) cross-modality few-shot learning [88, 89, 90] leverages semantic
information (e.g., word vectors) to augment the performance of visual tasks
and not among visual tasks only. The aforementioned approaches exclusively
address single-task learning per episode, while our work concurrently tackles
multiple heterogeneous and related tasks within each meta-training or meta-
test episode. Intuitive comparisons with the approaches are shown in Figure 6.

Figure 6: Differences from other episodic single-task setups. Each row
corresponds to an episode. Different color denotes different categories; the
same color with different shades represents different categories in the same
task. Episodic multi-task learning is orthogonal to these setups since it
concurrently tackles multiple heterogeneous and related tasks within each
episode.

#### Different roles of global and local latent variables.

In this paper, we introduce global latent representations and local latent
parameters within a hierarchical architecture. Each type of them plays a
distinct role in the proposed model: (1) Global latent representations provide
rich task-specific information during the inference of all local latent
parameters. This enables the model to generate task-specific decoders to
handle heterogeneous tasks in a single episode effectively. (2) Local latent
parameters with prediction-aware information constitute task-specific
decoders. Each local latent parameter reveals the knowledge corresponding to a
specific prediction across different tasks. This enhances the expressive power
of the proposed model. In practice, we observe that global latent
representations and local latent parameters complement each other when
performing predictions in meta-test episodes.

#### Advantages of the proposed hierarchical Bayes framework.

We summarize the advantages of the proposed framework. (1) A hierarchical
Bayesian framework with global and local latent variables yields a richer and
more complex latent space to mitigate the expressiveness bottleneck, thus
better parameterizing task-specific functions in stochastic processes. (2)
Global and local latent variables capture epistemic uncertainty in
representation and parameter levels, respectively, and show improved
performance in our experiments.

#### Roles of probabilistic HNPs and KL values in meta training.

The probabilistic HNPs encode the context as the heterogeneous prior and
reveal the uncertainty resulting from data insufficiency and the extent of
observations in tasks. Additionally, minimizing KL terms encourages priors
inferred from context sets to stay close to posteriors inferred from target
sets, guiding more efficient conditional generation. In training processes, we
observed the KL divergence value does not decrease to 0 after convergence,
e.g., KL values are in the order of e-1 on Office-home. This is also part of
traits in the NPs family, suggesting the approximate posterior and the
approximate prior encode different conditional information during the
generation of latent variables.

#### Real-world examples or benchmarks for episodic multi-task learning.

Episodic multi-task learning has several potential applications in the real
world, such as autonomous driving and robotic manipulations. In detail, the
autonomous driving system needs to deal with different and related sensor data
in an environment. However, the driving environment constantly changes along
with the weather, time, country, etc. Thus, fast adapting of the current
multi-tasker to new environments is challenging in this application and our
method can be a plausible solution for this challenge.

##  Appendix B Properties of Valid Exchangeable Stochastic Processes

Here, we further demonstrate that HNPs are valid stochastic processes, as
meeting the exchangeability and marginalization consistency conditions [12].
As stated in [12]: the conditions, including (finite) exchangeability and
marginalization, are sufficient to define a stochastic process with the help
of the Kolmogorov extension theorem. Here we follow the notations in the main
paper. For convenience, we provide the used symbols and the corresponding
descriptions in Table 10.

In this paper, we model the functional posterior distribution of the
stochastic process by approximating the joint distribution over all target
sets
$p(\mathbf{y}^{1:M}_{\tau}|\mathbf{x}^{1:M}_{\tau},\mathcal{C}^{1:M}_{\tau})$,
which is conditioned on all context samples $C^{1:M}_{\tau}$. To distinguish
the order set among different tasks, we use $n^{m}_{\tau}$ to denote the
number of target samples corresponding to a specific task in the episode
$\tau$. For simplicity, we omit the meta-knowledge $\omega$ and $\nu^{1:M}$ in
the formulations during the proof.

Table 10: Notations and their corresponding descriptions in this paper.

Notation Description $(\cdot)_{\tau}$ Variables correspond to an episode.
$(\cdot)^{m}$ Variables correspond to a single task. $\mathcal{I}_{\tau}^{m}$
A single task in the episode $\tau$, which is sampled from a specific task
distribution. $p(\mathcal{I}^{m})$ The specific task distribution. $M$ The
number of task distributions and the number of tasks in a single episode.
$\mathcal{I}_{\tau}^{1:M}$ All heterogeneous tasks in the episode $\tau$.
$\mathcal{C}$ A context set. $\mathcal{T}$ A target set. ${\bar{x}}$ A context
sample feature of the context set. ${{x}}$ A target sample feature of the
target set. $\mathbf{x}$ Set of all target sample features in the target set.
${\bar{y}}$ The ground truth of the context sample. ${{y}}$ The ground truth
of the target sample. $\mathbf{y}$ Set of the ground truth of all target
samples in the target set. $\mathcal{N}_{\mathcal{C}}$ The numbers of context
samples in the context set. $\mathcal{N}_{\mathcal{T}}$ The numbers of target
samples in the target set. $\mathbf{z}^{m}_{\tau}$ The introduced global
latent representation for a given task. $\mathbf{w}^{m}_{\tau,1:O}$ The
introduced local latent parameters for a given task.

###  B.1 Proof of Exchangeability Consistency

We now provide the proof of Exchangeability Consistency: the joint prediction
distribution is invariant to the permutation of the given multiple tasks and
the corresponding samples in each task.

######  Theorem B.1.

(Exchangability) For finite ${n^{*}_{\tau}}=\sum^{M}_{m=1}n^{m}_{\tau}$, if
${\pi}^{*}_{\tau}=\\{\pi^{m}_{\tau}\\}_{m=1}^{M}$ is a permutation of
$\\{1,\cdots,{n^{*}}\\}$ where $\pi^{m}_{\tau}$ is a permutation of the
corresponding order set $\\{1,\cdots,n^{m}_{\tau}\\}$, then:

$p({\pi^{*}_{\tau}}(\mathbf{y}^{1:M}_{\tau})|{\pi^{*}_{\tau}}(\mathbf{x}^{1:M}_{\tau}),\mathcal{C}^{1:M}_{\tau})=p(\mathbf{y}^{1:M}_{\tau}|\mathbf{x}^{1:M}_{\tau},\mathcal{C}^{1:M}_{\tau}),$

where
${\pi^{*}_{\tau}}(\mathbf{y}^{1:M}_{\tau}):=(\pi^{1}_{\tau}(\mathbf{y}^{1}_{\tau}),\cdots,\pi^{M}_{\tau}(\mathbf{y}^{M}_{\tau}))=({y}_{\tau,\pi^{*}_{\tau}(1)},\cdots,{y}_{\tau,\pi^{*}_{\tau}({n^{*}_{\tau}})})$
and
${\pi^{*}_{\tau}}(\\{\mathbf{x}^{1:M}_{\tau}\\}):=(\pi^{1}_{\tau}(\mathbf{x}^{1}_{\tau}),\cdots,\pi^{M}_{\tau}(\mathbf{x}^{M}_{\tau}))=({x}_{\tau,{\pi^{*}_{\tau}}(1)},\cdots,{x}_{\tau,{\pi^{*}_{\tau}}({n^{*}_{\tau}})})$.

###### Proof.

$\displaystyle
p({\pi^{*}_{\tau}}(\mathbf{y}^{1:M}_{\tau})|{\pi^{*}_{\tau}}(\mathbf{x}^{1:M}_{\tau}),\mathcal{C}^{1:M}_{\tau})$
$\displaystyle=\int\int
p({\pi^{*}_{\tau}}(\mathbf{y}^{1:M}_{\tau})|{\pi^{*}_{\tau}}(\mathbf{x}^{1:M}_{\tau}),\mathbf{w}^{1:M}_{\tau,1:O})\Big{(}\prod_{m=1}^{M}p(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}^{m}_{\tau},\mathcal{C}^{1:M}_{\tau})\Big{)}\Big{(}\prod_{m=1}^{M}p(\mathbf{z}^{m}_{\tau}|\mathcal{C}^{m}_{\tau})\Big{)}d\mathbf{w}^{1:M}_{\tau,1:O}d\mathbf{z}^{1:M}_{\tau}$
$\displaystyle=\int\int\Big{(}\prod_{i=1}^{{n^{*}_{\tau}}}p({y}_{\tau,{\pi^{*}_{\tau}}(i)}|{x}_{\tau,{\pi^{*}_{\tau}}(i)},\mathbf{w}^{1:M}_{\tau,1:O})\Big{)}\Big{(}\prod_{m=1}^{M}p(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}^{m}_{\tau},\mathcal{C}^{1:M}_{\tau})\Big{)}\Big{(}\prod_{m=1}^{M}p(\mathbf{z}^{m}_{\tau}|\mathcal{C}^{m}_{\tau})\Big{)}d\mathbf{w}^{1:M}_{\tau,1:O}d\mathbf{z}^{1:M}_{\tau}$
$\displaystyle=\int\int\prod_{m=1}^{M}\Big{\\{}\prod_{j=1}^{n^{m}_{\tau}}p({y}^{m}_{\tau,{\pi^{m}_{\tau}}(j)}|{x}^{m}_{\tau,{\pi^{m}_{\tau}}(j)},\mathbf{w}^{m}_{\tau,1:O})p(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}^{m}_{\tau},\mathcal{C}^{1:M}_{\tau})p(\mathbf{z}^{m}_{\tau}|\mathcal{C}^{m}_{\tau})\Big{\\}}d\mathbf{w}^{1:M}_{\tau,1:O}d\mathbf{z}^{1:M}_{\tau}$
$\displaystyle=\int\int\prod_{m=1}^{M}\Big{\\{}\prod_{j=1}^{n^{m}_{\tau}}p({y}^{m}_{\tau,j}|{x}^{m}_{\tau,j},\mathbf{w}^{m}_{\tau,1:O})p(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}^{m}_{\tau},\mathcal{C}^{1:M}_{\tau})p(\mathbf{z}^{m}_{\tau}|\mathcal{C}^{m}_{\tau})\Big{\\}}d\mathbf{w}^{1:M}_{\tau,1:O}d\mathbf{z}^{1:M}_{\tau}$
$\displaystyle=p(\mathbf{y}^{1:M}_{\tau}|\mathbf{x}^{1:M}_{\tau},\mathcal{C}^{1:M}_{\tau}).$

∎

###  B.2 Proof of Marginalization Consistency

We now aim to prove the Marginalization Consistency of the proposed model: if
marginalizing out a part of the target set in each task, the marginal
distribution remains the same as defined on the original target sets without
the marginalized part.

######  Theorem B.2.

(Marginalization) Given
${\hat{n}^{*}_{\tau}}=\sum_{m=1}^{M}\hat{n}^{m}_{\tau}$, where
$1\leq{\hat{n}^{*}_{\tau}}\leq{n^{*}_{\tau}}$ and for each task
$1\leq\hat{n}^{m}_{\tau}\leq{n}^{m}_{\tau}$, the consistency is:

$\int
p(\mathbf{y}^{1:M}_{\tau}|\mathbf{x}^{1:M}_{\tau},\mathcal{C}^{1:M}_{\tau})d(\mathbf{y}^{1:M}_{\tau})_{{\hat{n}^{*}_{\tau}}+1:{{n}^{*}_{\tau}}}=p((\mathbf{y}^{1:M}_{\tau})_{1:{\hat{n}^{*}_{\tau}}}|(\mathbf{x}^{1:M}_{\tau})_{1:{\hat{n}^{*}_{\tau}}},\mathcal{C}^{1:M}_{\tau}),$

where
$(\mathbf{y}^{1:M}_{\tau})_{1:{\hat{n}^{*}_{\tau}}}=((\mathbf{y}^{1}_{\tau})_{1:{\hat{n}^{1}_{\tau}}},\cdots,(\mathbf{y}^{M}_{\tau})_{1:{\hat{n}^{M}_{\tau}}})=(y_{\tau,1},\cdots,y_{\tau,\hat{n}^{*}_{\tau}})$
and
$(\mathbf{x}^{1:M}_{\tau})_{1:{\hat{n}^{*}_{\tau}}}=((\mathbf{x}^{1}_{\tau})_{1:{\hat{n}^{1}_{\tau}}},\cdots,(\mathbf{x}^{M}_{\tau})_{1:{\hat{n}^{M}_{\tau}}})=(x_{\tau,1},\cdots,x_{\tau,\hat{n}^{*}_{\tau}})$.

###### Proof.

$\displaystyle\int
p(\mathbf{y}^{1:M}_{\tau}|\mathbf{x}^{1:M}_{\tau},\mathcal{C}^{1:M}_{\tau})d(\mathbf{y}^{1:M}_{\tau})_{{\hat{n}^{*}_{\tau}}+1:{{n}^{*}_{\tau}}}$
$\displaystyle=\int\int\int
p(\mathbf{y}^{1:M}_{\tau}|\mathbf{x}^{1:M}_{\tau},\mathbf{w}^{1:M}_{\tau,1:O})\Big{(}\prod_{m=1}^{M}p(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}^{m}_{\tau},\mathcal{C}^{1:M}_{\tau})\Big{)}\Big{(}\prod_{m=1}^{M}p(\mathbf{z}^{m}_{\tau}|\mathcal{C}^{m}_{\tau})\Big{)}d\mathbf{w}^{1:M}_{\tau,1:O}d\mathbf{z}^{1:M}_{\tau}d(\mathbf{y}^{1:M}_{\tau})_{{\hat{n}^{*}_{\tau}}+1:{{n}^{*}_{\tau}}}$
$\displaystyle=\int\int\int\Big{(}\prod_{i=1}^{n^{*}_{\tau}}p(y_{\tau,i}|x_{\tau,i},\mathbf{w}^{1:M}_{\tau,1:O})\Big{)}\Big{(}\prod_{m=1}^{M}p(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}^{m}_{\tau},\mathcal{C}^{1:M}_{\tau})\Big{)}\Big{(}\prod_{m=1}^{M}p(\mathbf{z}^{m}_{\tau}|\mathcal{C}^{m}_{\tau})\Big{)}d\mathbf{w}^{1:M}_{\tau,1:O}d\mathbf{z}^{1:M}_{\tau}d(\mathbf{y}^{1:M}_{\tau})_{{\hat{n}^{*}_{\tau}}+1:{{n}^{*}_{\tau}}}$
$\displaystyle=\int\int\prod_{m=1}^{M}\Big{\\{}\int\prod_{j=1}^{n_{\tau}^{m}}p(y^{m}_{\tau,j}|x^{m}_{\tau,j},\mathbf{w}^{m}_{\tau,1:O})p(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}^{m}_{\tau},\mathcal{C}^{1:M}_{\tau})p(\mathbf{z}^{m}_{\tau}|\mathcal{C}^{m}_{\tau})d(\mathbf{y}^{m}_{\tau})_{{\hat{n}^{m}_{\tau}}+1:{{n}^{m}_{\tau}}}\Big{\\}}d\mathbf{w}^{1:M}_{\tau,1:O}d\mathbf{z}^{1:M}_{\tau}$
$\displaystyle=\int\int\prod_{m=1}^{M}\Big{\\{}\prod_{j=1}^{\hat{n}_{\tau}^{m}}p(y^{m}_{\tau,j}|x^{m}_{\tau,j},\mathbf{w}^{m}_{\tau,1:O})p(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}^{m}_{\tau},\mathcal{C}^{1:M}_{\tau})p(\mathbf{z}^{m}_{\tau}|\mathcal{C}^{m}_{\tau})$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
\int\prod_{j=\hat{n}_{\tau}^{m}+1}^{n_{\tau}^{m}}p(y^{m}_{\tau,j}|x^{m}_{\tau,j},\mathbf{w}^{m}_{\tau,1:O})d(\mathbf{y}^{m}_{\tau})_{{\hat{n}^{m}_{\tau}}+1:{{n}^{m}_{\tau}}}\Big{\\}}d\mathbf{w}^{1:M}_{\tau,1:O}d\mathbf{z}^{1:M}_{\tau}$
$\displaystyle=\int\int\prod_{m=1}^{M}\Big{\\{}\prod_{j=1}^{\hat{n}_{\tau}^{m}}p(y^{m}_{\tau,j}|x^{m}_{\tau,j},\mathbf{w}^{m}_{\tau,1:O})p(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}^{m}_{\tau},\mathcal{C}^{1:M}_{\tau})p(\mathbf{z}^{m}_{\tau}|\mathcal{C}^{m}_{\tau})\Big{\\}}d\mathbf{w}^{1:M}_{\tau,1:O}d\mathbf{z}^{1:M}_{\tau}$
$\displaystyle=p((\mathbf{y}^{1:M}_{\tau})_{1:{\hat{n}^{*}_{\tau}}}|(\mathbf{x}^{1:M}_{\tau})_{1:{\hat{n}^{*}_{\tau}}},\mathcal{C}^{1:M}_{\tau}).$

∎

##  Appendix C Tractable and Scalable Optimization

For the proposed HNPs, it is intractable to obtain the true joint posterior
$p(\mathbf{w}_{\tau,1:O}^{1:M},\mathbf{z}_{\tau}^{1:M}|\mathcal{T}_{\tau}^{1:M};\omega,\nu^{1:M})$
for each episode. Thus, we employ variational inference to optimize the
designed model by approximating the true joint posterior in each episode. To
do so, we introduce the variational joint posterior distribution:

$\displaystyle
q_{\theta,\phi}(\mathbf{w}_{\tau,1:O}^{1:M},\mathbf{z}_{\tau}^{1:M}|\mathcal{T}_{\tau}^{1:M};\omega,\nu^{1:M})=\prod_{m=1}^{M}q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega),$
(11)

where $q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})$ and
$q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega)$
are variational posteriors of their corresponding latent variables. We note
that variational posteriors are inferred from the target sets that are
available in the meta-training stage. The variational posteriors are
parameterized as diagonal Gaussian distributions [104]. The inference networks
$\theta$ and $\phi$ are shared by the variational posteriors and their
corresponding priors, following the protocol of vanilla NPs [12].

###  C.1 Derivation of Approximate ELBO for HNPs

By incorporating the variational posteriors in Eq. (11) into the modeling of
HNPs in the main paper, we can derive the approximate ELBO
$L_{\rm{HNPs}}(\omega,\nu^{1:M},\theta,\phi)$ as follows:

$\displaystyle\log
p(\mathcal{T}_{\tau}^{1:M}|\mathcal{C}_{\tau}^{1:M};\omega,\nu^{1:M})$ (12)
$\displaystyle=\sum_{m=1}^{M}\log
p(\mathcal{T}_{\tau}^{m}|\mathcal{C}_{\tau}^{1:M};\omega,\nu^{m})$
$\displaystyle=\sum_{m=1}^{M}\bigg{\\{}\log\int\Big{\\{}\int
p(\mathbf{y}_{\tau}^{m}|\mathbf{x}_{\tau}^{m},\mathbf{w}_{\tau,1:O}^{m})p_{\phi}(\mathbf{w}_{\tau,1:O}^{m}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega)d\mathbf{w}^{m}_{\tau,1:O}\Big{\\}}p_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{C}_{\tau}^{m};\nu^{m})d\textbf{z}_{\tau}^{m}\bigg{\\}}$
$\displaystyle=\sum_{m=1}^{M}\bigg{\\{}\log\int\Big{\\{}\int
p(\mathbf{y}_{\tau}^{m}|\mathbf{x}_{\tau}^{m},\mathbf{w}_{\tau,1:O}^{m})p_{\phi}(\mathbf{w}_{\tau,1:O}^{m}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega)\frac{q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega)}{q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega)}d\mathbf{w}^{m}_{\tau,1:O}\Big{\\}}$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
p_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{C}_{\tau}^{m};\nu^{m})\frac{q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})}{q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})}d\textbf{z}_{\tau}^{m}\bigg{\\}}$
$\displaystyle\geq\sum_{m=1}^{M}\bigg{\\{}\mathbb{E}_{q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})}\Big{\\{}\log\int
p(\mathbf{y}_{\tau}^{m}|\mathbf{x}_{\tau}^{m},\mathbf{w}_{\tau,1:O}^{m})p_{\phi}(\mathbf{w}_{\tau,1:O}^{m}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega)\frac{q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega)}{q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega)}d\mathbf{w}^{m}_{\tau,1:O}\Big{\\}}$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
-\mathbb{D}_{\rm{KL}}[q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})||p_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{C}_{\tau}^{m};\nu^{m})]\bigg{\\}}$
$\displaystyle\geq\sum_{m=1}^{M}\bigg{\\{}\mathbb{E}_{q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})}\Big{\\{}\mathbb{E}_{q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega)}[\log
p(\mathbf{y}_{\tau}^{m}|\mathbf{x}_{\tau}^{m},\mathbf{w}^{m}_{\tau,1:O})]$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
-\mathbb{D}_{\rm{KL}}[q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{T}_{\tau}^{1:M};\omega)||p_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|\mathbf{z}_{\tau}^{m},\mathcal{C}_{\tau}^{1:M};\omega)]\Big{\\}}$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
-\mathbb{D}_{\rm{KL}}[q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})||p_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{C}_{\tau}^{m};\nu^{m})]\bigg{\\}}=L_{\rm{HNPs}}(\omega,\nu^{1:M},\theta,\phi).$

In general, when the variational joint posterior is flexible enough, the
posterior approximation gap between the variational posterior and the
intractable true posterior can be reduced to an arbitrarily small quantity
[73]. In this case, maximizing the approximate ELBO increases the overall log
likelihood in the proposed model accordingly. We construct task-specific
decoders in a data-driven way by inferring local latent parameters
$\mathbf{w}^{m}_{1:O}$ from all context sets and the meta-knowledge. This
enables our model to amortize the training cost of each task-specific decoder,
further reducing the model’s over-fitting behaviors for episodic multi-task
learning.

###  C.2 Meta-Training Objective

In practice, we consider the loss function as the negative approximate ELBO of
HNPs as given in Eq. (12). By adopting Monte Carlo sampling [104, 105], the
meta-training objective for each episode is:

$\displaystyle-
L_{\rm{HNPs}}(\omega,\nu^{1:M},\theta,\phi)\approx\sum_{m=1}^{M}\bigg{\\{}\frac{1}{N_{z}}\sum_{i=1}^{N_{z}}\Big{\\{}\frac{1}{N_{w}}\sum_{j=1}^{N_{w}}[-\log
p(\mathbf{y}_{\tau}^{m}|\mathbf{x}_{\tau}^{m},{\mathbf{w}^{m}_{\tau,1:O}}^{(j)})]$
(13)
$\displaystyle+\mathbb{D}_{\rm{KL}}[q_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|{\mathbf{z}_{\tau}^{m}}^{(i)},\mathcal{T}_{\tau}^{1:M};\omega)||p_{\phi}(\mathbf{w}^{m}_{\tau,1:O}|{\mathbf{z}_{\tau}^{m}}^{(i)},\mathcal{C}_{\tau}^{1:M};\omega)]\Big{\\}}$
$\displaystyle+\mathbb{D}_{\rm{KL}}[q_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{T}_{\tau}^{m};\nu^{m})||p_{\theta}(\mathbf{z}_{\tau}^{m}|\mathcal{C}_{\tau}^{m};\nu^{m})]\bigg{\\}},$

where ${\mathbf{z}_{\tau}^{m}}^{(i)}$ and ${\mathbf{w}^{m}_{\tau,1:O}}^{(j)}$
are sampled from their corresponding variational posteriors. ${N_{z}}$ and
$N_{w}$ are the number of Monte Carlo samples for ${\mathbf{z}_{\tau}^{m}}$
and ${\mathbf{w}^{m}_{\tau,1:O}}$, respectively.

###  C.3 Meta-Test Prediction

At the meta-test stage, we perform predictions with the learned model on the
target sets for a new episode ${\tau^{*}}$, which involves the prior
distributions of global and local latent variables. We again approximate the
predictive distribution with Monte Carlo estimates:

$\displaystyle
p(\mathcal{T}_{\tau^{*}}^{1:M}|\mathcal{C}_{\tau^{*}}^{1:M};\omega,\nu^{1:M})\approx\prod_{m=1}^{M}\bigg{\\{}\frac{1}{N_{z}}\sum_{i=1}^{N_{z}}\frac{1}{N_{w}}\sum_{j=1}^{N_{w}}p(\mathbf{y}_{\tau^{*}}^{m}|\mathbf{x}_{\tau^{*}}^{m},{\mathbf{w}_{\tau^{*},1:O}^{m}}^{(j)})\bigg{\\}},$
(14)

where ${\mathbf{w}_{\tau^{*},1:O}^{m}}^{(j)}\sim
p_{\phi}(\mathbf{w}^{m}_{\tau^{*},1:O}|{\mathbf{z}_{\tau^{*}}^{m}}^{(i)},\mathcal{C}_{\tau^{*}}^{1:M};\omega)$
and ${\mathbf{z}_{\tau^{*}}^{m}}^{(i)}\sim
p_{\theta}(\mathbf{z}_{\tau^{*}}^{m}|\mathcal{C}_{\tau^{*}}^{m};\nu^{m})$.
Here the Monte Carlo samples follow their corresponding prior distributions
since the target sets are unavailable during the meta-test.

##  Appendix D More Experimental Details

###  D.1 Transformer-structured Inference Modules in Regression Scenarios

Here we present the implementation details of transformer-structured inference
module $\theta$ in regression scenarios. In a single episode $\tau$, the
module $\theta$ encodes the task-specific information into each refined task-
specific token $\nu^{m}$, and then infers the prior distribution or the
variational posterior distribution for the global latent representation
$\mathbf{z}^{m}_{\tau}$. For episodic multi-task regression, the local latent
parameters $\mathbf{w}_{\tau,1:O}^{m}$ construct a task-specific regressor
during inference. We assume that the output of the decoder follows a Gaussian
distribution for regression tasks. Thus, $\mathbf{w}^{m}_{\tau,1:O}$ are
instantiated as parameters for generating the mean and variance of
predictions.

###  D.2 Backbone and Training Details

Following the protocol of [10], we apply the pre-trained deep model as the
backbone for the proposed method and baselines to extract the input features
under the episodic multi-task classification setup. To be specific, we adopt
VGGnet [106] for Office-Home, and its feature size is $4096$. We take ResNet18
[107] for DomainNet with input size $512$. In practice, we train our method
and baselines by the Adam optimizer [108] using an NVIDIA Tesla V100 GPU. The
learning rate is initially set as $1e-4$ and decreases with a factor of $0.5$
every $3,000$ iterations.

##  Appendix E Algorithm of the proposed HNPs

Input : $M$ distinct and relevant task distributions $p(\mathcal{I}^{1:M})$,
numbers of Monte Carlo samples $N_{z}$ and $N_{w}$, learning rates $\alpha$.

Output : Meta-trained transformer inference module $\theta$ and $\phi$,
learnable tokens $\omega_{1:O}$ and $\nu^{1:M}$.

Initialize transformer inference module $\\{\theta,\nu^{1:M}\\}$ ;

Initialize transformer inference module $\\{\phi,\omega_{1:O}\\}$ ;

while  _Meta-Training not Completed_ do

// sample a meta-training episode indexed with $\tau$.

for  _$m=1$ to $M$_ do

Sample a task $\mathcal{I}^{m}_{\tau}\sim p(\mathcal{I}^{m})$, which shares
the same target space $\mathcal{Y}_{\tau}$ with other tasks in the episode;

Sample a context set $\mathcal{C}^{m}_{\tau}$ and a target set
$\mathcal{T}^{m}_{\tau}$ for the task $\mathcal{I}^{m}_{\tau}$;

end for

// infer hierarchical latent variables.

for  _$m=1$ to $M$_ do

Apply transformer inference module $\theta$ to infer prior and variational
posterior distributions over $\mathbf{z}^{m}_{\tau}$ as Eq. (5), Eq. (6), and
Eq. (7) in the main paper;

Draw $N_{z}$ samples from the variational posterior,
$\\{{\mathbf{z}^{m}_{\tau}}^{(i)}\\}_{i=1}^{N_{z}}$;

for  _$i=1$ to $N_{z}$_ do

for  _$o=1$ to $O$_ do

Apply transformer inference module $\phi$ to infer prior and variational
posterior distributions over $\mathbf{w}^{m}_{\tau,o}$ as Eq. (8), Eq. (9),
and Eq. (10) in the main paper;

Draw $N_{w}$ samples from the variational posterior,
$\\{{\mathbf{w}^{m}_{\tau,o}}^{(j)}\\}_{j=1}^{N_{w}}$;

end for

end for

end for

// optimize the objective.

Compute predictive distributions and minimize the empirical objective in Eq.
(13) ;

Update $\theta$, $\phi$, $\omega_{1:O}$ and $\nu^{1:M}$ with learning rate
$\alpha$.

end while

Algorithm 1 Meta-training phase of HNPs.

Input : Meta-trained transformer inference module $\theta$ and $\phi$, learned
tokens $\omega_{1:O}$ and $\nu^{1:M}$, numbers of Monte Carlo samples $N_{z}$
and $N_{w}$.

Output : Prediction results.

while  _Meta-Test not Completed_ do

// given a meta-test episode indexed with $\tau^{*}$.

Collect the context sets of all tasks in the episode,
$\mathcal{C}^{1:M}_{\tau^{*}}$.

for  _$m=1$ to $M$_ do

Apply transformer inference module $\theta$ to infer prior distributions over
$\mathbf{z}^{m}_{\tau^{*}}$ as Eq. (5), Eq. (6), and Eq. (7) in the main
paper;

Draw $N_{z}$ samples from the prior distribution,
$\\{{\mathbf{z}^{m}_{\tau^{*}}}^{(i)}\\}_{i=1}^{N_{z}}$;

for  _$i=1$ to $N_{z}$_ do

for  _$o=1$ to $O$_ do

Apply transformer inference module $\phi$ to infer the prior distribution over
$\mathbf{w}^{m}_{\tau^{*},o}$ as Eq. (8), Eq. (9), and Eq. (10) in the main
paper;

Draw $N_{w}$ samples from the prior distribution,
$\\{{\mathbf{w}^{m}_{\tau^{*},o}}^{(j)}\\}_{j=1}^{N_{w}}$;

end for

end for

end for

Perform predictions on the target sets in Eq. (14) ;

end while

Algorithm 2 Meta-test phase of HNPs. Table 11: Performance comparisons on Office-Home under the Xtask5way1shot setup during inference. Number of tasks | 1 | 2 | 3 | 4
---|---|---|---|---
Average accuracy | 64.33 $\pm$ 0.85 | 70.75 $\pm$ 0.67 | 74.95 $\pm$ 0.57 | 76.29 $\pm$ 0.51
Table 12: Performance comparisons under the 4task20way1shot and 20way4shot setups. Methods | 4task20way1shot | 20way4shot
---|---|---
MAML | $34.29\pm 0.19$ | $37.23\pm 0.25$
Proto.Net | $32.72\pm 0.18$ | $37.12\pm 0.22$
Ours | $51.82\pm 0.23$ | -
Table 13: Performance comparisons with more recent baselines. Methods | 1-shot | 5-shot
---|---|---
MTL-bridge [76] | $64.31\pm 0.55$ | $75.10\pm 0.51$
MLTI [109] | $70.69\pm 0.73$ | $79.59\pm 0.58$
Ours | $76.29\pm 0.51$ | $80.80\pm 0.42$
Table 14: Performance comparisons on the Office31 dataset.

Methods ERM Proto.Net CNPs NPs TNP-D Ours Average Accuracy $63.53\pm 0.71$
$64.54\pm 0.64$ $49.02\pm 0.74$ $40.52\pm 0.75$ $69.69\pm 0.87$ $71.89\pm
0.52$

##  Appendix F More Experimental Results under Episodic Multi-Task Setup

###  F.1 Effects of More “Tasks”

To investigate the effects of more "tasks" in the episodic multi-task setup,
we perform experiments on Office-Home under the Xway5way1shot setup by
gradually increasing tasks during inference. Table 11 shows that the average
accuracy increases with more tasks. The main reason is that more tasks can
provide richer transferable information. Our model benefits from the positive
transfer among tasks, and thus obtaining higher performance gain from more
tasks.

###  F.2 4task20way1shot v.s. 20way4shot

To show the effectiveness of the proposed method, we conduct experiments with
the 20-way 4-shot setup, which needs to mix samples from all tasks in one
episode. As shown in Table 12, MAML and Proto.Net perform better under the
20-way 4-shot but cannot outperform our method. The main reason is that our
method can better handle distribution shifts among tasks by exploring task-
relatedness rather than simply mixing them together.

###  F.3 Comparisons with More Recent Works

To compare the proposed method with more recent works, we perform experiments
on Office-Home under the 4task5way1shot and 4task5way5shot setups. We provide
some brief descriptions of two recent works as follows:

[76] theoretically addresses the conclusion that MTL methods are powerful and
efficient alternatives to GBML for meta-learning applications. However, our
method inherits the advantages of multi-task learning and meta-learning:
simultaneously generalizing meta-knowledge from past to new episodes and
exploiting task-relatedness across heterogeneous tasks in every single
episode. Thus, our method is more suitable for solving the data-insufficiency
problem.

[109] augments the task set in meta-learning through interpolation. Our method
fully utilizes several observed tasks in a single episode rather than
generating additional tasks.

Table 13 shows that our method significantly outperforms other baselines with
severely insufficient data, such as 1-shot. This is consistent with the
conclusion obtained in the main paper.

###  F.4 Comparisons on Another Benchmark Dataset

We validate the performance of methods on the Office31 dataset [110, 111]
under the 3task5way1shot setup. This dataset contains 31 object categories in
three domains: Amazon, DSLR, and Webcam. Table 14 shows our method outperforms
baseline methods, demonstrating our model’s effectiveness in addressing the
data insufficiency under the episodic setup.

##  Appendix G More Experimental Results under Conventional Multi-Task Setup

To show comparisons with existing multi-task models, which are designed for
conventional multi-task learning, we extend the proposed HNPs to conventional
multi-task learning settings for both regression and classification tasks by
considering only one episode during training and inference. Under conventional
multi-task settings (MIMO), we investigate their effectiveness in exploring
shared knowledge when limited tasks and samples are available during training.

###  G.1 Conventional Multi-Task Regression

#### Dataset and Settings.

We show the effectiveness of HNPs for conventional multi-task regression. We
design experiments for rotation angle estimation on the Rotated MNIST dataset
[112]. Each task is an angle estimation problem for a digit, and different
tasks corresponding to individual digits are related because they share the
same rotation angle space. Each image is rotated by $0^{\circ}$ through
$90^{\circ}$ in intervals of $10^{\circ}$, where the rotation angle is the
regression target. We randomly choose $0.1\%$ training samples per task per
angle as the training set.

We use the average of normalized mean squared errors (NMSE) of all tasks as
the performance measurement. The lower NMSE, the better the performance. We
provide $95$% confidence intervals for the errors from five runs. Descriptions
of baselines can be found in Section G.2.

#### Results and Discussions.

The experimental results are summarized in Table 15. The proposed HNPs
outperform other counterpart methods by yielding a lower mean error. This
demonstrates the effectiveness of HNPs in capturing task-relatedness for
improved regression performance.

Table 15: Conventional multi-task regression (normalized mean squared errors)
for rotation angle estimation.

Methods Average NMSE STL .215 $\pm$.001 VSTL .224 $\pm$.004 BMTL .118
$\pm$.003 VBMTL .121 $\pm$.003 LearnToBranch [48] .109 $\pm$.002 VMTL [11]
.110 $\pm$.003 HNPs .103 $\pm$.001

###  G.2 Conventional Multi-Task Classification

#### Datasets and Settings.

Office-Caltech [113] contains ten categories shared between Office-31 [110]
and Caltech-256 [114]. One task uses data from Caltech-256, and the other
tasks use data from Office-31, whose images were collected from three distinct
domains/tasks, namely Amazon, Webcam and DSLR. There are $8\sim 151$ samples
per category per task and $2,533$ images. ImageCLEF [10], the benchmark for
the ImageCLEF domain adaptation challenge, contains $12$ common categories
shared by four public datasets/tasks: Caltech-256, ImageNet ILSVRC 2012,
Pascal VOC 2012, and Bing. There are $2,400$ images in total. Office-Home [91]
mentioned in the main paper is also used under this setting.

We adopt the standard evaluation protocols [10] for multi-task classification
datasets. We randomly select $5\%$, $10\%$ and $20\%$ labeled data for
training, which correspond to about 3, 6 and 12 samples per category per task,
respectively. In this case, each task has insufficient training data for
building a reliable classifier without overfitting. The average accuracy of
all tasks is used for measuring the overall performance. We again provide
$95$% confidence intervals for the errors from five runs.

#### Alternatives Methods.

We conduct a thorough comparison with alternative multi-task learning models.
Single-task learning (STL) is implemented by task-specific feature extractors
and predictors without knowledge sharing among tasks. Basic multi-task
learning (BMTL) shares feature extractors and adds task-specific predictors.
We also define variational extensions of single-task learning (VSTL) and basic
multi-task learning (VBMTL), which treat predictors as latent variables [11].
For a fair comparison, all the baseline methods mentioned above share the same
architecture of the feature extractor and the train-test splits. We also
compare the proposed HNPs to representative multi-task models. MTL-Uncertainty
[24], MRN [10], LearnToBranch [48] are deep MTL methods, employing deep neural
networks to construct information-sharing mechanisms for tasks. TCGBML [35],
MTVIB [41] and VMTL [11] are probabilistic MTL methods, applying Bayes
frameworks to model the relationships among tasks.

Table 16: Classification performance (average accuracy) on Office-Home,
Office-Caltech and ImageCLEF.

Office-Home Office-Caltech ImageCLEF Methods 5% 10% 20% 5% 10% 20% 5% 10% 20%
STL 49.2 $\pm$0.2 58.3 $\pm$0.1 64.9 $\pm$0.1 88.6 $\pm$0.3 90.7 $\pm$0.2 92.4
$\pm$0.3 62.6 $\pm$0.2 69.7 $\pm$0.3 76.2 $\pm$0.3 VSTL 51.1 $\pm$0.1 60.2
$\pm$0.2 65.8 $\pm$0.2 89.0 $\pm$0.2 91.1 $\pm$0.2 93.4 $\pm$0.3 64.9 $\pm$0.3
70.8 $\pm$0.3 77.2 $\pm$0.2 BMTL 50.4 $\pm$0.1 59.5 $\pm$0.1 65.6 $\pm$0.1
89.5 $\pm$0.3 92.3 $\pm$0.2 93.1 $\pm$0.1 65.7 $\pm$0.4 72.0 $\pm$0.3 76.8
$\pm$0.3 VBMTL 51.3 $\pm$0.1 60.9 $\pm$0.1 67.0 $\pm$0.2 90.8 $\pm$0.6 93.2
$\pm$0.2 93.5 $\pm$0.1 67.1 $\pm$0.3 73.0 $\pm$0.7 78.0 $\pm$0.2 MTL-
Uncertainty[24] 51.8 $\pm$0.1 57.2 $\pm$0.2 66.8 $\pm$0.2 91.2 $\pm$0.3 93.8
$\pm$0.2 94.7 $\pm$0.3 74.6 $\pm$0.2 76.9 $\pm$0.3 79.2 $\pm$0.3 MRN [10] 57.4
$\pm$0.1 63.4 $\pm$0.2 67.1 $\pm$0.1 93.4 $\pm$0.2 94.8 $\pm$0.3 95.1 $\pm$0.1
73.7 $\pm$0.4 75.8 $\pm$0.2 79.7 $\pm$0.3 LearnToBranch [48] 38.3 $\pm$0.5
51.5 $\pm$0.3 62.2 $\pm$0.4 74.6 $\pm$0.9 80.4 $\pm$1.2 89.9 $\pm$0.8 51.7
$\pm$0.9 62.6 $\pm$0.8 71.6 $\pm$0.4 TCGBML[35] 52.8 $\pm$0.1 60.0 $\pm$0.2
68.7 $\pm$0.2 91.8 $\pm$0.1 95.0 $\pm$0.2 95.1 $\pm$0.1 73.9 $\pm$0.3 76.5
$\pm$0.4 79.3 $\pm$0.4 MTVIB [41] 49.9 $\pm$0.2 55.3 $\pm$0.1 66.2 $\pm$0.1
91.1 $\pm$0.3 94.1 $\pm$0.3 95.0 $\pm$0.2 74.0 $\pm$0.4 77.3 $\pm$0.3 78.9
$\pm$0.5 VMTL [11] 58.3 $\pm$0.1 65.0 $\pm$0.0 69.2 $\pm$0.2 93.8 $\pm$0.1
95.3 $\pm$0.0 95.2 $\pm$0.1 76.2 $\pm$0.3 77.9 $\pm$0.2 80.2 $\pm$0.1 HNPs
60.0 $\pm$0.1 66.2 $\pm$0.2 70.9 $\pm$0.2 94.6 $\pm$0.1 95.4 $\pm$0.1 95.8
$\pm$0.1 76.4 $\pm$0.1 79.5 $\pm$0.1 80.9 $\pm$0.1

#### Results and Discussions.

We provide more comprehensive comparisons on Office-Home, Office-Caltech and
ImageCLEF in Table 16. The best results are marked in bold. Our HNPs achieve
competitive and even better performance on conventional multi-task
classification datasets with different train-test splits. VSTL and VBMTL
perform better than STL and BMTL, demonstrating the benefits of Bayes
frameworks. Compared with multi-task probabilistic baselines, including VBMTL,
TCGBML, MTVIB and VMTL, our HNPs can model more complex functional
distributions with more powerful priors by inferring both representations and
parameters for predictive functions.

Compared with VMTL [11], which neglects the hierarchical architecture of
latent variables, the proposed HNPs show better performance. This demonstrates
that by modeling the complex dependencies between heterogeneous context sets
within the hierarchical Bayes framework, HNPs explore task-relatedness better.
The hierarchical Bayes framework enables our model to explore the relevant
knowledge even in the presence of distribution shifts among tasks.

##  Appendix H Application to Brain Image Segmentation

To show that HNPs have the potential to be helpful in settings other than
categorization and regression, we apply the proposed HNPs to brain image
segmentation.

#### Dataset and Settings.

We adopt a brain image dataset [115] with lower-grade gliomas collected from
$110$ patients. The number of images varies among patients from $20$ to $88$.
The goal is to segment the tumor in each brain image by predicting its
contour.

Figure 7: Segmentation results by HNPs (bottom row) and U-Net (upper row). The
ground truth contours are in green, and the predicted ones are in red. The
numbers are DSC scores computed against the ground truth. HNPs can predict
contours closer to the ground truth ones, indicating the advantages of
exploring spatial context information for image segmentation.

We reformulate the segmentation task as a pixel-wise regression problem, where
each pixel corresponds to a regression task to predict the probability of this
pixel belonging to the tumor. In doing so, the spatial correlation and
dependency among pixels are effectively modeled by capturing task-relatedness.
For the task $m$, we define $\Omega_{m}$ as a local region centered at the
spatial position $m$, which provides the local context information. In this
case, the region centered at the pixel provides the local context information.
Each task incorporates the knowledge provided by related tasks into the
context of the predictive function. This offers an effective way to model the
long-range interdependence of pixels in one image. In the implementation, we
use U-Net [116] as the backbone and append our model to the last layer.

#### Results and Discussions.

The proposed HNPs surpass the baseline U-Net by 0.8% (91.2% v.s. 90.6%) in
terms of dice similarity coefficients (DSC) for the overall validation set. We
provide the predicted contours by HNPs (bottom row) and the U-Net (upper row)
in Figure 7, where the green outline corresponds to the ground truth and the
red to the segmentation output. This figure shows that HNPs predict contours
closer to the ground truth. The results demonstrate the advantages of HNPs in
exploring spatial-relatedness for medical segmentation.