# Addressing Parameter Choice Issues in Unsupervised Domain Adaptation by
Aggregation

Marius-Constantin Dinu1,2 Markus Holzleitner1 Maximilian Beck1 &Duc Hoan
Nguyen5 Andrea Huber1 Hamid Eghbal-zadeh1 Bernhard A. Moser4 &Sergei V.
Pereverzyev5 Sepp Hochreiter1,3 Werner Zellinger5

1ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes
Kepler University Linz
2Dynatrace Research
3Institute of Advanced Research in Artificial Intelligence
4Software Competence Center Hagenberg
5Johann Radon Institute for Computational and Applied Mathematics, Austrian
Academy of Sciences

###### Abstract

We study the problem of choosing algorithm hyper-parameters in unsupervised
domain adaptation, i.e., with labeled data in a source domain and unlabeled
data in a target domain, drawn from a different input distribution. We follow
the strategy to compute several models using different hyper-parameters, and,
to subsequently compute a linear aggregation of the models. While several
heuristics exist that follow this strategy, methods are still missing that
rely on thorough theories for bounding the target error. In this turn, we
propose a method that extends weighted least squares to vector-valued
functions, e.g., deep neural networks. We show that the target error of the
proposed algorithm is asymptotically not worse than twice the error of the
unknown optimal aggregation. We also perform a large scale empirical
comparative study on several datasets, including text, images,
electroencephalogram, body sensor signals and signals from mobile phones. Our
method111Large scale benchmark experiments are available at
https://github.com/Xpitfire/iwa;
<EMAIL_ADDRESS><EMAIL_ADDRESS>outperforms deep embedded
validation (DEV) and importance weighted validation (IWV) on all datasets,
setting a new state-of-the-art performance for solving parameter choice issues
in unsupervised domain adaptation with theoretical error guarantees. We
further study several competitive heuristics, all outperforming IWV and DEV on
at least five datasets. However, our method outperforms each heuristic on at
least five of seven datasets.

##  1 Introduction

The goal of unsupervised domain adaptation is to learn a model on unlabeled
data from a target input distribution using labeled data from a different
source distribution (Pan & Yang, 2010; Ben-David et al., 2010). If this goal
is achieved, medical diagnostic systems can successfully be trained on
unlabeled images using labeled images with a different modality (Varsavsky et
al., 2020; Zou et al., 2020); segmentation models for natural images can be
learned using only labeled data from computer simulations Peng et al. (2018);
natural language models can be learned from unlabeled biomedical abstracts by
means of labeled data from financial journals (Blitzer et al., 2006);
industrial quality inspection systems can be learned on unlabeled data from
new products using data from related products (Jiao et al., 2019; Zellinger et
al., 2020).

However, missing target labels combined with distribution shift makes
parameter choice a hard problem (Sugiyama et al., 2007; You et al., 2019;
Saito et al., 2021; Zellinger et al., 2021; Musgrave et al., 2021). Often, one
ends up with a sequence of models, e.g., originating from different hyper-
parameter configurations (Ben-David et al., 2007; Saenko et al., 2010; Ganin
et al., 2016; Long et al., 2015; Zellinger et al., 2017; Peng et al., 2019).
In this work, we study the problem of constructing an optimal aggregation
using all models in such a sequence. Our main motivation is that the error of
such an optimal aggregation is clearly smaller than the error of the best
single model in the sequence.

Although methods with mathematical error guarantees have been proposed to
select the best model in the sequence (Sugiyama et al., 2007; Kouw et al.,
2019; You et al., 2019; Zellinger et al., 2021), methods for learning
aggregations of the models are either heuristics or their theory guarantees
are limited by severe assumptions (cf. Wilson & Cook (2020)). Typical
aggregation approaches are (a) to learn an aggregation on source data only
(Nozza et al., 2016), (b) to learn an aggregation on a set of (unknown)
labeled target examples (Xia et al., 2013; Dai et al., 2007; III & Marcu,
2006; Duan et al., 2012), (c) to learn an aggregation on target examples
(pseudo-)labeled based on confidence measures of the given models (Zhou et
al., 2021; Ahmed et al., 2022; Sun, 2012; Zou et al., 2018; Saito et al.,
2017), (d) to aggregate the models based on data-structure specific
transformations (Yang et al., 2012; Ha & Youn, 2021), and, (e) to use specific
(possibly not available) knowledge about the given models, such as information
obtained at different time-steps of its gradient-based optimization process
(French et al., 2018; Laine & Aila, 2017; Tarvainen & Valpola, 2017;
Athiwaratkun et al., 2019; Al-Stouhi & Reddy, 2011) or the information that
the given models are trained on different (source) distributions (Hoffman et
al., 2018; Rakshit et al., 2019; Xu et al., 2018; Kang et al., 2020; Zhang et
al., 2015). One problem shared among all methods mentioned above is that they
cannot guarantee a small error, even if the sample size grows to infinity. See
Figure 1 for a simple illustrative example.

Figure 1: Unsupervised domain adaptation problem (Shimodaira, 2000; Sugiyama
et al., 2007; You et al., 2019). Left: Source distribution (solid) and target
distribution (dashed). Right: A sequence of different linear models (dashed)
is used to find the optimal linear aggregation of the models (solid). Model
selection methods (Sugiyama et al., 2007; Kouw et al., 2019; You et al., 2019;
Zellinger et al., 2021) cannot outperform the best single model in the
sequence, confidence values as used in Zou et al. (2018) are not available,
and, approaches based on averages or tendencies of majorities of models (Saito
et al., 2017) suffer from a high fraction of large-error-models in the
sequence. In contrast, our approach (dotted-dashed) is nearly optimal. In
addition, the model computed by our method provably approaches the optimal
linear aggregation for increasing sample size. For further details on this
example we refer to Section C in the Supplementary Material.

In this work, we propose (to the best of our knowledge) the first algorithm
for computing aggregations of vector-valued models for unsupervised domain
adaptation with target error guarantees. We extend the importance weighted
least squares algorithm (Shimodaira, 2000) and corresponding recently proposed
error bounds (Gizewski et al., 2022) to linear aggregations of vector-valued
models. The importance weights are the values of an estimated ratio between
target and source density evaluated at the examples. Every method for density-
ratio estimation can be used as a basis for our approach, e.g. Sugiyama et al.
(2012); Kanamori et al. (2012) and references therein. Our error bound proves
that the target error of the computed aggregation is asymptotically at most
twice the target error of the optimal aggregation.

In addition, we perform extensive empirical evaluations on several datasets
with academic data (Transformed Moons), text data (Amazon Reviews (Blitzer et
al., 2006)), images (MiniDomainNet (Peng et al., 2019; Zellinger et al.,
2021)), electroencephalography signals (Sleep-EDF (Eldele et al., 2021;
Goldberger et al., 2000)), body sensor signals (UCI-HAR (Anguita et al.,
2013), WISDM (Kwapisz et al., 2011)), and, sensor signals from mobile phones
and smart watches (HHAR (Stisen et al., 2015)).

We compute aggregations of models obtained from different hyper-parameter
settings of 11 domain adaptation methods (e.g., DANN (Ganin et al., 2016) and
Deep-Coral Sun & Saenko (2016)). Our method sets a new state of the art for
methods with theoretical error guarantees, namely importance weighted
validation (IWV) (Sugiyama et al., 2007) and deep embedded validation (DEV)
(Kouw et al., 2019), on all datasets. We also study (1) classical least
squares aggregation on source data only, (2) majority voting on target
predictions, (3) averaging over model confidences, and (4) learning based on
pseudo-labels. All of these heuristics outperform IWV and DEV on at least five
of seven datasets, which is a result of independent interest. In contrast, our
method outperforms each heuristic on at least five of seven datasets.

Our main contributions are summarized as follows:

  * •

We propose the (to the best of our knowledge) first algorithm for ensemble
learning of vector-valued models in (single-source) unsupervised domain
adaptation that satisfies a non-trivial target error bound.

  * •

We prove that the target error of our algorithm is asymptotically (for
increasing sample sizes) at most twice the target error of the unknown optimal
aggregation.

  * •

We outperform IWV and DEV, and therefore set a new state-of-the-art
performance for re-solving parameter choice issues under theoretical target
error guarantees.

  * •

We describe four heuristic baselines which all outperform IWV and DEV on at
least five of seven datasets. Our method outperforms each heuristic on at
least five of seven datasets.

  * •

Our method tends to be more stable than others w.r.t. adding inaccurate models
to the given sequence of models.

##  2 Related Work

It is well known that aggregations of models in an ensemble often outperform
individual models (Dong et al., 2020; Goodfellow et al., 2016). Traditional
ensemble methods that have shown the advantage of aggregation are Boosting
(Schapire, 1990; Breiman, 1998), Bootstrap Aggregating (bagging) (Breiman,
1994; 1996a) and Stacking (Wolpert, 1992; Breiman, 1996b). For example,
averages of multiple models pre-trained on data from a distribution different
from the target one have recently been shown to achieve state-of-the-art
performance on ImageNet (Wortsman et al., 2022) and their good generalization
properties can be related to flat minima (Hochreiter & Schmidhuber, 1994;
1997). However, most such methods don’t take into account a present
distribution shift.

Although some ensemble learning methods exist, which take into account a
present distribution shift, in contrast to our work, they are either relying
on labeled target data (Nozza et al., 2016; Xia et al., 2013; III & Marcu,
2006; Dai et al., 2007; Mayr et al., 2016), are restricted by fixing the
aggregation weights to be the same (Razar & Samothrakis, 2019), make
assumptions on the models in the sequence or the corresponding process for
learning the models (Yang et al., 2012; Ha & Youn, 2021; French et al., 2018;
Laine & Aila, 2017; Tarvainen & Valpola, 2017; Athiwaratkun et al., 2019; Al-
Stouhi & Reddy, 2011; Hoffman et al., 2018; Rakshit et al., 2019; Xu et al.,
2018; Kang et al., 2020; Zhang et al., 2015), or, learn an aggregation based
on the heuristic approach of (pseudo-)labeling some target data based on
confidence measures of models in the sequence (Zhou et al., 2021; Ahmed et
al., 2022; Sun, 2012; Zou et al., 2018; Saito et al., 2017). Another crucial
difference of all methods above is that none of these methods can guarantee a
small target error in the general setting (distribution shift, vector valued
models, different classes, single source domain) described above, even if the
sample size grows to infinity.

Another branch of research are methods which aim at selecting the best model
in the sequence. Although, such methods with error bounds have been proposed
for the general setting above (Sugiyama et al., 2007; You et al., 2019;
Zellinger et al., 2021), they cannot overcome a limited performance of the
best model in the given sequence (cf. Figure 1 and Section 6 in the
Supplementary Material of Zellinger et al. (2021)). In contrast, our method
can outperform the best model in the sequence, and our empirical evaluations
show that this is indeed the case in practical examples. A recent kernel-based
algorithm for univariate regression, that is similar to ours, can be found in
Gizewski et al. (2022). However, in contrast to Gizewski et al. (2022), our
method allows a much more general form of vector-valued models which are not
necessarily obtained from regularized kernel least squares, and, can therefore
be applied to practical deep learning tasks.

Our work employs technical tools developed in Caponnetto & De Vito (2007;
2005). In fact, we extend Caponnetto & De Vito (2007; 2005) to deal with
importance weighted least squares. Finally, it is important to note Huang et
al. (2006), where a core Lemma of our proofs is proposed.

##  3 Aggregation by Importance Weighted Least Squares

This section gives a summary of the main problem of this paper and our
approach. For detailed assumptions and proofs, we refer to Section A of the
Supplementary Material.

##### Notation and Setup

Let $\mathcal{X}\subset\mathbb{R}^{d_{1}}$ be a compact input space and
$\mathcal{Y}\subset\mathbb{R}^{d_{2}}$ be a compact label space with inner
product $\left\langle.,.\right\rangle_{\mathcal{Y}}$ such that for the
associated norm $\left\lVert y\right\rVert_{\mathcal{Y}}\leq y_{0}$ holds for
all $y\in\mathcal{Y}$ and some $y_{0}>0$. Following Ben-David et al. (2010),
we consider two datasets: A source dataset
$(\mathbf{x},{\mathbf{y}})=((x_{1},y_{1}),\ldots,(x_{n},y_{n}))\in\left(\mathcal{X}\times\mathcal{Y}\right)^{n}$
independently drawn according to some source distribution (probability
measure) $p$ on $\mathcal{X}\times\mathcal{Y}$ and an unlabeled target dataset
$\mathbf{x}^{\prime}=(x_{1}^{\prime},\ldots,x_{m}^{\prime})\in\mathcal{X}^{m}$
with elements independently drawn according to the marginal distribution222The
existence of the conditional probability density $q(y|x)$ with
$q(x,y)=q(y|x)q_{\mathcal{X}}(x)$ is guaranteed by the fact that
$\mathcal{X}\times\mathcal{Y}$ is Polish, i.e., a separable and complete
metric space, c.f. Dudley (2002, Theorem 10.2.2.). $q_{\mathcal{X}}$ of some
target distribution $q$ on $\mathcal{X}\times\mathcal{Y}$. The marginal
distribution of $p$ on $\mathcal{X}$ is analogously denoted as
$p_{\mathcal{X}}$. We further denote by
$\mathcal{R}_{q}(f)=\int_{\mathcal{X}\times\mathcal{Y}}\left\lVert
f(x)-y\right\rVert_{\mathcal{Y}}^{2}\mathop{}\\!\mathrm{d}q(x,y)$ the expected
target risk of a vector valued function $f:\mathcal{X}\to\mathcal{Y}$ w.r.t.
the least squares loss.

##### Problem

Given a set $f_{1},\ldots,f_{l}:\mathcal{X}\to\mathcal{Y}$ of models, the
labeled source sample $(\mathbf{x},{\mathbf{y}})$ and the unlabeled target
sample $\mathbf{x}^{\prime}$, the problem considered in this work is to find a
model $f:\mathcal{X}\to\mathcal{Y}$ with a minimal target error
$\mathcal{R}_{q}(f)$.

##### Main Assumptions

We rely (a) on the covariate shift assumption that the source conditional
distribution $p(y|x)$ equals the target conditional distribution $q(y|x)$,
and, (b) on the bounded density ratio assumption that there is a function
$\beta:\mathcal{X}\to[0,B]$ with $B>0$ such that
$\mathop{}\\!\mathrm{d}q_{\mathcal{X}}(x)=\beta(x)\mathop{}\\!\mathrm{d}p_{\mathcal{X}}(x)$.

##### Approach

Our goal is to compute the linear aggregation $f=\sum_{i=1}^{l}c_{i}f_{i}$ for
${c_{1},\ldots,c_{l}\in\mathbb{R}}$ with minimal squared target risk
$\mathcal{R}_{q}\left(\sum_{i=1}^{l}c_{i}f_{i}\right)$. Our approach relies on
the fact that

$\displaystyle\operatorname*{arg\,min}_{c_{1},\ldots,c_{l}\in\mathbb{R}}\mathcal{R}_{q}\left(\sum_{i=1}^{l}c_{i}f_{i}\right)=\operatorname*{arg\,min}_{c_{1},\ldots,c_{l}\in\mathbb{R}}\int_{\mathcal{X}}\left\lVert\sum_{i=1}^{l}c_{i}f_{i}(x)-f_{q}(x)\right\rVert_{\mathcal{Y}}^{2}\mathop{}\\!\mathrm{d}q_{\mathcal{X}}(x)$
(1)

for the regression function
$f_{q}(x)=\int_{\mathcal{Y}}y\mathop{}\\!\mathrm{d}q(y|x)$333$\mathcal{Y}$-valued
integrals are defined in the sense of Lebesgue-Bochner., see e.g. Cucker &
Smale (2002, Proposition 1). Unfortunately, the right hand side of Eq. (1)
contains information about labels $f_{q}(x)$ which are not given in our
setting of unsupervised domain adaptation. However, borrowing an idea from
importance sampling, it is possible to estimate Eq. (1). More precisely, from
the covariate shift assumption we get
$f_{p}(x)=\int_{\mathcal{Y}}y\mathop{}\\!\mathrm{d}p(y|x)=f_{q}(x)$ and we can
use the bounded density ratio $\beta$ to obtain

$\displaystyle\operatorname*{arg\,min}_{c_{1},\ldots,c_{l}\in\mathbb{R}}\mathcal{R}_{q}\left(\sum_{i=1}^{l}c_{i}f_{i}\right)=\operatorname*{arg\,min}_{c_{1},\ldots,c_{l}\in\mathbb{R}}\int_{\mathcal{X}}\beta(x)\left\lVert\sum_{i=1}^{l}c_{i}f_{i}(x)-f_{p}(x)\right\rVert_{\mathcal{Y}}^{2}\mathop{}\\!\mathrm{d}p_{\mathcal{X}}(x)$
(2)

which extends importance weighted least squares (Shimodaira, 2000; Kanamori et
al., 2009) to linear aggregations $\sum_{i=1}^{l}c_{i}f_{i}$ of vector-valued
functions $f_{1},\ldots,f_{l}$. The unique minimizer of Eq. (2) can be
approximated based on available data analogously to classical least squares
estimation as detailed in Algorithm 1. In the following, we call Algorithm 1
Importance Weighted Least Squares Linear Aggregation (IWA).

##### Relation to Model Selection

The optimal aggregation
$f^{*}:=\operatorname*{arg\,min}_{c_{1},\ldots,c_{l}\in\mathbb{R}}\mathcal{R}_{q}\left(\sum_{i=1}^{l}c_{i}f_{i}\right)$
defined in Eq. (2) is clearly better than any single model selection since

$\displaystyle\mathcal{R}_{q}(f^{*})=\min_{c_{1},\ldots,c_{l}\in\mathbb{R}}\mathcal{R}_{q}\left(\sum_{i=1}^{l}c_{i}f_{i}\right)\leq\min_{c_{1},\ldots,c_{l}\in\\{0,1\\}}\mathcal{R}_{q}\left(\sum_{i=1}^{l}c_{i}f_{i}\right)\leq\min_{f_{1},\ldots,f_{l}}\mathcal{R}_{q}(f_{i}).$
(3)

However, the optimal aggregation $f^{*}$ cannot be computed based on finite
datasets and the next logical questions are about the accuracy of the
approximation $\widetilde{f}$ in Algorithm 1.

Input : Set $f_{1},\ldots,f_{l}:\mathcal{X}\to\mathcal{Y}$ of models, labeled
source sample $(\mathbf{x},{\mathbf{y}})$ and unlabeled target sample
$\mathbf{x}^{\prime}$.

Output : Linear aggregation
$\widetilde{f}=\sum_{i=1}^{l}\widetilde{c}_{i}f_{i}$ with weights
$\widetilde{c}=(\widetilde{c}_{1},\ldots,\widetilde{c}_{l})\in\mathbb{R}^{l}$.

Step 1 _Use unlabeled samples $\mathbf{x}$ and $\mathbf{x}^{\prime}$ to
approximate density ratio
$\frac{\mathop{}\\!\mathrm{d}q_{\mathcal{X}}}{\mathop{}\\!\mathrm{d}p_{X}}$ by
some function $\beta:\mathcal{X}\to[0,B]$ using a classical algorithm, e.g.
Sugiyama et al. (2012). missing_

Step 2 _Compute weight vector $\widetilde{c}=\widetilde{G}^{-1}\widetilde{g}$
with empirical Gram matrix $\widetilde{G}$ and vector $\widetilde{g}$ defined
by _ $\displaystyle\widetilde{G}=\left(\frac{1}{m}\sum_{k=1}^{m}\left\langle
f_{i}(x_{k}^{\prime}),f_{j}(x_{k}^{\prime})\right\rangle_{\mathcal{Y}}\right)_{i,j=1}^{l}\quad\quad\widetilde{g}=\left(\frac{1}{n}\sum_{k=1}^{n}\beta(x_{k})\left\langle
y_{k},f_{i}(x_{k})\right\rangle_{\mathcal{Y}}\right)_{i=1}^{l}.$ __

Return : Linear aggregation
$\widetilde{f}=\sum_{i=1}^{l}\widetilde{c}_{i}f_{i}$.

Algorithm 1 Importance Weighted Least Squares Linear Aggregation (IWA).

##  4 Target Error Bound for Algorithm 1

Let us start by introducing some further notation: $L^{2}(p)$ refers to the
Lebesgue-Bochner space of functions from $\mathcal{X}$ to $\mathcal{Y}$,
associated to a measure $p$ on $\mathcal{X}$ with corresponding inner product
$\left\langle.,.\right\rangle_{L^{2}(p)}$ (this space basically consists of
all $\mathcal{Y}$-valued functions whose $\mathcal{Y}$-norms are square
integrable with respect to the given measure $p$). Moreover, let us introduce
the (positive semi-definite) Gram matrix $G=\left(\left\langle
f_{i},f_{j}\right\rangle_{L^{2}(q_{\mathcal{X}})}\right)_{i,j=1}^{l}$ and the
vector $\overline{g}=\left(\left\langle\beta
f_{p},f_{i}\right\rangle_{L^{2}(p_{\mathcal{X}})}\right)_{i=1}^{l}$. We can
assume that $G$ is invertible (and thus positive definite), since otherwise
some models are too similar to others and can be withdrawn from consideration
(see Section D). Next, we recall that the minimizer of Eq. (2) is
$c^{*}=(c_{1}^{*},\ldots,c_{l}^{*})=G^{-1}\overline{g}$, see Lemma 4.

However, neither $G$ nor the vector $\overline{g}$ is accessible in practice,
because there is no access to the target measure $q_{\mathcal{X}}$. Driven by
the law of large numbers we try to approximate them by averages over our given
data and therefore arrive at the formulas for $\widetilde{G}$ and
$\widetilde{g}$ given in Algorithm 1. This leads to the approximation
$\widetilde{f}$. Up to this point, we were only considering an intuitive
perspective on the problem setting, therefore, we will now formally discuss
statements on the distance between the model $\widetilde{f}$ and the optimal
linear model $f^{*}=\sum_{i=1}^{l}c_{i}^{*}f_{i}$, measured in terms of target
risks, and how this distance behaves with increasing sample sizes. This is
what we attempt with our main result:

######  Theorem 1.

With probability $1-\delta$ it holds that

$\displaystyle\mathcal{R}_{q}(\tilde{f})-\mathcal{R}_{q}(f_{q})\leq
2\left(\mathcal{R}_{q}\left(f^{*}\right)-\mathcal{R}_{q}(f_{q})\right)+C\left(\log\frac{1}{\delta}\right)(n^{-1}+m^{{-1}})$
(4)

for some coefficient $C>0$ not depending on $m,n$ and $\delta$, and
sufficiently large $m$ and $n$.

Before we give an outline of the proof (see Section A), let us briefly comment
on the main message of Algorithm 1. Observe, that (Cucker & Smale, 2002,
Proposition 1) $\mathcal{R}_{q}(f)-\mathcal{R}_{q}(f_{q})=\left\lVert
f-f_{q}\right\rVert_{L^{2}(q_{\mathcal{X}})}^{2}$ can be interpreted as the
total target error made by Algorithm 1, sometimes called excess risk. Indeed,
in the deterministic setting of labeling functions, $f_{q}$ equals the target
labeling function and the excess risk equals the target error of Ben-David et
al. (2010). Eq. (4) compares this error for the aggregation $\widetilde{f}$,
computed by Algorithm 1, to the error for the optimal aggregation $f^{*}$.
Note that the error of the optimal aggregation $f^{*}$ is unavoidable in the
sense that it is determined by the decision of searching for linear
aggregations of $f_{1},\ldots,f_{l}$ only. However, if the models
$f_{1},\ldots,f_{l}$ are sufficiently different, then this error can be
expected to be small. Theorem 1 tells us that the error of $\widetilde{f}$
approaches the one of $f^{*}$ with increasing target and source sample size.
The rate of convergence is at least linear. Finally, we emphasize that Theorem
1 does not take into account the error of the density-ratio estimation. We
refer to the recent work Gizewski et al. (2022), who, for the first time,
included such error in the analysis of importance weighted least squares.

Let us now give a brief outline for the proof of Theorem 1. One key part
concerns the existence of a Hilbert space $\mathcal{H}$ with associated inner
product $\left\langle.,.\right\rangle_{\mathcal{H}}$ (a reproducing kernel
space of functions from $\mathcal{X}\to\mathcal{Y}$) which contains all given
models $f_{1},\ldots,f_{l}$ and the regression function $f_{q}=f_{p}$. The
space $\mathcal{H}$ can be constructed from any given models that are bounded
and continuous functions. Furthermore, Algorithm 1 does not need any knowledge
of $\mathcal{H}$, which is a modeling assumption only needed for the proofs,
so that we can apply many arguments developed in Caponnetto & De Vito (2007;
2005). $\mathcal{H}$ is also not necessarily generated by a prescribed kernel
such as Gaussian or linear kernel, and, no further smoothness assumption is
required, see Sections A and B in the Supplementary Material.

Moreover, in this setting one can express the excess risk as follows:
$\mathcal{R}_{q}\left(f\right)-\mathcal{R}_{q}(f_{q})=\left\lVert
A(f-f_{q})\right\rVert_{\mathcal{H}}^{2}$ for some bounded linear operator
$A:\mathcal{H}\to\mathcal{H}$. This also allows us to formulate the entries of
$G$ and $\bar{g}$ in terms of the inner product
$\left\langle.,.\right\rangle_{\mathcal{H}}$ instead. Using properties related
to the operators that appear in the construction of $\mathcal{H}$, in
combination with Hoeffding-like concentration bounds in Hilbert spaces and
bounds that measure, e.g., the deviation between empirical averages in source
and target domain (as done in Gretton et al. (2006, Lemma 4)), we can quantify
differences between the entries of $G$ and $\widetilde{G}$ (and $\bar{g}$ and
$\widetilde{g}$ respectively) in terms of $n$, $m$ and $\delta$. This leads to
Eq. (4).

##  5 Empirical Evaluations

Figure 2: Top: Mean classification accuracy (y-axis) of our method (IWA),
source-only regression (SOR), deep embedded validation (DEV) and individual
models (green: source accuracy, orange: target accuracy) used in the
aggregation for the HHAR dataset (Stisen et al., 2015) over 3 seeds. The
individual models (x-axis) are trained with DIRT (Shu et al., 2018) for
different hyper-parameter choices. Bottom: Scaled Aggregation weights (y-axis)
for individual models (x-axis) computed by IWA, SOR and DEV (average over 3
seeds). Instead of searching for the best model in the sequence, IWA
effectively uses all models in the sequence and obtains a performance not
reachable by any procedure selecting only one model.

We now empirically evaluate the performance of our approach compared to
classical ensemble learning baselines and state-of-the-art model selection
methods. Therefore, we structure our empirical evaluation as follows. First,
we outline our experimental setup for unsupervised domain adaptation and
introduce all domain adaptation methods for our analysis. Second, we describe
the ensemble learning and model selection baselines, and third, we present the
datasets used for our experiments. We then conclude with our results and a
detailed discussion thereof.

###  5.1 Experimental Setup

To assess the performance of our ensemble learning Algorithm 1 IWA, we perform
numerous experiments with different domain adaptation algorithms on different
datasets. By changing the hyper-parameters of each algorithm, we obtain, as
results of applying these algorithms, sequences of models. The goal of our
method is to find optimal models based on combinations of candidates from each
sequence. As domain adaptation algorithms, we consider the AdaTime benchmark
suite, and run our experiments on language, image, text and time-series data.
This suite comprises a collection of $11$ domain adaptation algorithms. We
follow their evaluation setup and apply the following algorithms: Adversarial
Spectral Kernel Matching (AdvSKM) (Liu & Xue, 2021), Deep Domain Confusion
(DDC) (Tzeng et al., 2014), Correlation Alignment via Deep Neural Networks
(Deep-Coral) (Sun et al., 2017), Central Moment Discrepancy (CMD) (Zellinger
et al., 2017), Higher-order Moment Matching (HoMM) (Chen et al., 2020),
Minimum Discrepancy Estimation for Deep Domain Adaptation (MMDA) (Rahman et
al., 2020), Deep Subdomain Adaptation (DSAN) (Zhu et al., 2021), Domain-
Adversarial Neural Networks (DANN) (Ganin et al., 2016), Conditional
Adversarial Domain Adaptation (CDAN) (Long et al., 2018), A DIRT-T Approach to
Unsupervised Domain Adaptation (DIRT) (Shu et al., 2018) and Convolutional
deep Domain Adaptation model for Time-Series data (CoDATS) (Wilson et al.,
2020). In addition to the sequence of models, IWA requires an estimate of the
density ratio between source and target domain. To compute this quantity we
follow (Bickel et al., 2007) and (You et al., 2019, Section 4.3), and, train a
classifier discriminating between source and target data. The output of this
classifier is then used to approximate the density ratio denoted as $\beta$ in
Algorithm 1. Overall, to compute the results in our tables we trained $16680$
models over approximately a timeframe of $1500$ GPU/hours using computation
resources of NVIDIA P100 $16$GB GPUs.

For example, consider the top plot of Figure 2, where we compare the
performance of Algorithm 1 to deep embedded validation (DEV) (You et al.,
2019), a heuristics baseline source-only regression (SOR, see Section 5.2) and
each individual model in the sequence. The bottom plot shows the the scaled
aggregation weights, i.e. how much each individual model contributes to the
aggregated prediction of IWA, DEV, and SOR. In this example, the given
sequence of models is obtained from applying the algorithm proposed in Shu et
al. (2018) with different hyper-parameter choices to the Heterogeneity Human
Activity Recognition dataset (Stisen et al., 2015). See Section D.3 in the
Supplementary Material for the exact hyper-parameter values.

###  5.2 Baselines

As representatives for the most prominent methods discussed in Section 1, we
compare our method, IWA, to ensemble learning methods that use linear
regression and majority voting as heuristic for model aggregation, and, model
selection methods with theoretical error guarantees.

Heuristic Baselines The first baseline is majority voting on target data
(TMV). It aggregates the predictions of all models by counting the overall
class predictions and selects the class with the maximum prediction count as
ensemble output. In addition, we implement three heuristic baselines which
aggregate the vector-valued output, i.e. probabilities, of all classifiers
using weights learned via linear regression. The final ensemble prediction is
then made by selecting the class with the highest probability. The three
heuristic regression baselines differ in the input used for the performed
regression. Source-only regression (SOR) trains a regression model on
classifier predictions (of the given models) and labels from the source domain
only. Target majority voting regression (TMR) uses the same voting procedure
as explained above to generate pseudo-labels on the target domain, which are
then further used to train a linear regression model. In contrast, target
confidence average regression (TCR) selects the highest average class
probability over all classifiers to pseudo-label the target samples, which is
then used for training the linear regression model.

Baselines with Theoretical Error Guarantees We compare IWA to the model
selection methods importance weighted validation (IWV) (Sugiyama et al., 2007)
and deep embedded validation (DEV) (You et al., 2019), which select models
according to their (importance weighted) target risk. Both methods assume the
knowledge of an estimated density ratio between target and source domains. In
our experiments we follow Bickel et al. (2007); You et al. (2019) and estimate
this ratio, by using a classifier that discriminates between source and target
domain (see Supplementary Material Section D for more details).

###  5.3 Datasets

We evaluate the previously mentioned methods according to a diverse set of
datasets, including language, image and time-series data. All datasets have a
train, evaluation and test split, with results only presented on the held-out
test sets. For additional details we refer to Appendix C and D.

TransformedMoons This specific form of twinning moons is based on Zellinger et
al. (2021). The source domain consists of two-dimensional input data points
and their transformations to two opposing moon-shaped forms.

MiniDomainNet is a reduced version of DomainNet-2019 (Peng et al., 2019)
consisting of six different image domains (Quickdraw, Real, Clipart, Sketch,
Infograph, and Painting). In particular, MiniDomainNet (Zellinger et al.,
2021) reduces the number of classes of DomainNet-2019 to the top-five largest
representatives in the training set of each class across all six domains.

AmazonReviews is based on Blitzer et al. (2006) and consists of text reviews
from four domains: books, DVDs, electronics, and kitchen appliances. Reviews
are encoded in feature vectors of bag-of-words unigrams and bigrams with
binary labels indicating the rankings. From the four categories we obtain
twelve domain adaptation tasks where each category serves once as source
domain and once as target domain.

UCI-HAR The _Human Activity Recognition_ (Anguita et al., 2013) dataset from
the UC Irvine Repository contains data from three motion sensors
(accelerometer, gyroscope and body-worn sensors) gathered using smartphones
from $30$ different subjects. It classifies their activities in several
categories, namely, walking, walking upstairs, downstairs, standing, sitting,
and lying down.

WISDM (Kwapisz et al., 2011) is a class-imbalanced dataset variant from
collected accelerometer sensors, including GPS data, from $29$ different
subjects which are performing similar activities as in the UCI-HAR dataset.

HHAR The _Heterogeneity Human Activity Recognition_ (Stisen et al., 2015)
dataset investigate sensor-, device- and workload-specific heterogeneities
using $36$ smartphones and smartwatches, consisting of $13$ different device
models from four manufacturers.

Sleep-EDF The _Sleep Stage Classification_ time-series setting aims to
classify the electroencephalography (EEG) signals into five stages i.e., Wake
(W), Non-Rapid Eye Movement stages (N1, N2, N3), and Rapid Eye Movement (REM).
Analogous to Ragab et al. (2022); Eldele et al. (2021), we adopt the Sleep-
EDF-20 dataset obtained from PhysioBank (Goldberger et al., 2000), which
contains EEG readings from $20$ healthy subjects.

We rely on the AdaTime benchmark suite (Ragab et al., 2022) in most
evaluations. The four time-series datasets above are originally included
there. We extend AdaTime to support the other discussed datasets as well, and
extend its domain adaptation methods.

###  5.4 Results

We separate the applied methods into two groups, namely _heuristic_ and
methods with _theoretical error guarantees_. All tables show accuracies of
source-only (SO) and target-best (TB) models, where source-only denotes
training without domain adaptation and target-best the best performing model
obtained among all parameter settings. We highlight in bold the performance of
the best performing method with theoretical error guarantees, and in italic
the best performing heuristic. See Table 1 for results. Please find the full
tables in the Supplementary Material Section D.

Outperformance of theoretically justified methods: On all datasets, our method
outperforms IWV and DEV, setting a new state of the art for solving parameter
choice issues under theoretical guarantees.

Outperformance of heuristics: It is interesting to note that each heuristic
outperforms IWV and DEV on at least five of seven datasets. Moreover, every
heuristic outperforms the (average) target best model (TB) in at least two
cases, making it impossible for any model selection method to win in these
cases. These facts highlight the quality of the predictions of our chosen
heuristics. However, each heuristic is outperformed by our method on at least
five of seven datasets.

Information in aggregation weights and robustness w.r.t. inaccurate models: It
is interesting to observe that, in contrast to the other heuristic aggregation
baselines, the aggregation weights $c_{1},\ldots,c_{l}$ of our method tend to
be larger for accurate models, see Section D.5. Another result is that our
method tends to be less sensitive to a high number of inaccurate models than
the baselines, see Section D.6. This serves as another reason for its high
empirical performance.

##  6 Conclusion and Future Work

We present a constructive theory-based method for approaching parameter choice
issues in the setting of unsupervised domain adaptation. Its theoretical
approach relies on the extension of weighted least squares to vector-valued
functions. The resulting aggregation method distinguishes itself by a wide
scope of admissible model classes without strong assumptions, e.g. support
vector machines, decision trees and neural networks. A broad empirical
comparative study on benchmark datasets for language, images, body sensor
signals and handy signals, underpins the theory-based optimality claim. It is
left for future research to further refine the theory and its estimates, e.g.,
by exploiting concentration bounds from Gretton et al. (2006) or advanced
density ratio estimators from Sugiyama et al. (2012).

Table 1: Mean and standard deviation (after $\pm$) of target classification
accuracy on Amazon Reviews, Sleep-EDF, UCI-HAR, HHAR and WISDM datasets over
three different random initialization of model weights and several domain
adaptation tasks.

| Amazon Reviews
---|---
|  | Heuristic | Theoretical error guarantees |
Method | SO | TMV | TMR | TCR | SOR | IWV | DEV | IWA (ours) | TB
HoMM | $0.769(\pm 0.009)$ | $0.777(\pm 0.010)$ | $0.778(\pm 0.010)$ | ${\it 0.778(\pm 0.011)}$ | $0.777(\pm 0.010)$ | $0.765(\pm 0.011)$ | $0.766(\pm 0.011)$ | ${\bf 0.778(\pm 0.010)}$ | $0.769(\pm 0.012)$
AdvSKM | $0.766(\pm 0.012)$ | ${\it 0.780(\pm 0.009)}$ | $0.779(\pm 0.010)$ | $0.779(\pm 0.008)$ | $0.778(\pm 0.011)$ | $0.769(\pm 0.012)$ | $0.766(\pm 0.012)$ | ${\bf 0.780(\pm 0.009)}$ | $0.770(\pm 0.012)$
DIRT | $0.764(\pm 0.009)$ | $0.786(\pm 0.008)$ | $0.786(\pm 0.010)$ | $0.786(\pm 0.008)$ | ${\it 0.800(\pm 0.008)}$ | $0.778(\pm 0.022)$ | $0.773(\pm 0.056)$ | ${\bf 0.787(\pm 0.008)}$ | $0.786(\pm 0.009)$
DDC | $0.766(\pm 0.012)$ | $0.779(\pm 0.010)$ | ${\it 0.780(\pm 0.009)}$ | $0.779(\pm 0.010)$ | $0.778(\pm 0.010)$ | $0.767(\pm 0.017)$ | $0.768(\pm 0.011)$ | ${\bf 0.780(\pm 0.010)}$ | $0.770(\pm 0.013)$
CMD | $0.767(\pm 0.012)$ | $0.791(\pm 0.009)$ | $0.792(\pm 0.009)$ | $0.789(\pm 0.010)$ | ${\it 0.792(\pm 0.010)}$ | $0.765(\pm 0.015)$ | $0.710(\pm 0.015)$ | ${\bf 0.794(\pm 0.009)}$ | $0.785(\pm 0.009)$
MMDA | $0.767(\pm 0.011)$ | $0.787(\pm 0.011)$ | $0.785(\pm 0.010)$ | $0.785(\pm 0.010)$ | ${\it 0.787(\pm 0.012)}$ | $0.769(\pm 0.011)$ | $0.766(\pm 0.010)$ | ${\bf 0.787(\pm 0.011)}$ | $0.782(\pm 0.011)$
CoDATS | $0.766(\pm 0.013)$ | $0.795(\pm 0.009)$ | $0.793(\pm 0.010)$ | $0.794(\pm 0.012)$ | ${\it 0.799(\pm 0.010)}$ | $0.779(\pm 0.016)$ | $0.773(\pm 0.020)$ | ${\bf 0.796(\pm 0.009)}$ | $0.791(\pm 0.015)$
Deep-Coral | $0.766(\pm 0.012)$ | ${\it 0.784(\pm 0.009)}$ | $0.783(\pm 0.009)$ | $0.783(\pm 0.009)$ | $0.782(\pm 0.009)$ | $0.769(\pm 0.016)$ | $0.769(\pm 0.037)$ | ${\bf 0.785(\pm 0.009)}$ | $0.776(\pm 0.013)$
CDAN | $0.767(\pm 0.012)$ | ${\it 0.788(\pm 0.010)}$ | $0.787(\pm 0.009)$ | $0.787(\pm 0.010)$ | $0.787(\pm 0.011)$ | $0.775(\pm 0.011)$ | $0.776(\pm 0.014)$ | ${\bf 0.788(\pm 0.010)}$ | $0.777(\pm 0.011)$
DANN | $0.767(\pm 0.012)$ | $0.796(\pm 0.010)$ | $0.792(\pm 0.010)$ | $0.793(\pm 0.010)$ | ${\it 0.800(\pm 0.011)}$ | $0.776(\pm 0.011)$ | $0.778(\pm 0.012)$ | ${\bf 0.797(\pm 0.009)}$ | $0.798(\pm 0.012)$
DSAN | $0.769(\pm 0.009)$ | $0.796(\pm 0.009)$ | $0.792(\pm 0.009)$ | $0.791(\pm 0.010)$ | ${\it 0.800(\pm 0.010)}$ | $0.779(\pm 0.012)$ | $0.763(\pm 0.017)$ | ${\bf 0.795(\pm 0.009)}$ | $0.789(\pm 0.012)$
Avg. | $0.767(\pm 0.011)$ | $0.787(\pm 0.009)$ | $0.786(\pm 0.010)$ | $0.786(\pm 0.010)$ | ${\it 0.789(\pm 0.010)}$ | $0.772(\pm 0.014)$ | $0.764(\pm 0.019)$ | ${\bf 0.788(\pm 0.009)}$ | $0.781(\pm 0.012)$

| Sleep-EDF
---|---
|  | Heuristic | Theoretical error guarantees |
Method | SO | TMV | TMR | TCR | SOR | IWV | DEV | IWA (ours) | TB
HoMM | $0.676(\pm 0.036)$ | $0.722(\pm 0.017)$ | $0.719(\pm 0.023)$ | $0.718(\pm 0.021)$ | ${\it 0.724(\pm 0.032)}$ | $0.726(\pm 0.046)$ | $0.678(\pm 0.035)$ | ${\bf 0.747(\pm 0.025)}$ | $0.715(\pm 0.047)$
AdvSKM | $0.665(\pm 0.058)$ | $0.708(\pm 0.023)$ | $0.712(\pm 0.027)$ | $0.712(\pm 0.032)$ | ${\it 0.718(\pm 0.030)}$ | $0.703(\pm 0.069)$ | $0.692(\pm 0.038)$ | ${\bf 0.722(\pm 0.025)}$ | $0.706(\pm 0.054)$
DIRT | $0.656(\pm 0.058)$ | $0.743(\pm 0.009)$ | $0.745(\pm 0.012)$ | ${\it 0.748(\pm 0.019)}$ | $0.742(\pm 0.031)$ | $0.679(\pm 0.038)$ | $0.686(\pm 0.066)$ | ${\bf 0.749(\pm 0.010)}$ | $0.728(\pm 0.037)$
DDC | $0.646(\pm 0.035)$ | $0.717(\pm 0.029)$ | ${\it 0.721(\pm 0.037)}$ | $0.712(\pm 0.031)$ | $0.695(\pm 0.020)$ | $0.694(\pm 0.056)$ | $0.666(\pm 0.031)$ | ${\bf 0.724(\pm 0.012)}$ | $0.704(\pm 0.031)$
CMD | $0.653(\pm 0.057)$ | ${\it 0.740(\pm 0.022)}$ | $0.736(\pm 0.016)$ | $0.723(\pm 0.020)$ | $0.709(\pm 0.015)$ | $0.716(\pm 0.052)$ | $0.640(\pm 0.068)$ | ${\bf 0.729(\pm 0.018)}$ | $0.725(\pm 0.053)$
MMDA | $0.650(\pm 0.051)$ | ${\it 0.736(\pm 0.014)}$ | $0.727(\pm 0.021)$ | $0.723(\pm 0.018)$ | $0.714(\pm 0.028)$ | $0.704(\pm 0.033)$ | $0.660(\pm 0.034)$ | ${\bf 0.745(\pm 0.031)}$ | $0.715(\pm 0.042)$
CoDATS | $0.672(\pm 0.084)$ | $0.738(\pm 0.029)$ | ${\it 0.739(\pm 0.036)}$ | $0.736(\pm 0.030)$ | $0.723(\pm 0.039)$ | $0.683(\pm 0.090)$ | $0.690(\pm 0.107)$ | ${\bf 0.744(\pm 0.021)}$ | $0.715(\pm 0.045)$
Deep-Coral | $0.643(\pm 0.049)$ | $0.716(\pm 0.018)$ | ${\it 0.717(\pm 0.028)}$ | $0.712(\pm 0.027)$ | $0.694(\pm 0.032)$ | $0.700(\pm 0.053)$ | $0.675(\pm 0.077)$ | ${\bf 0.713(\pm 0.021)}$ | $0.702(\pm 0.070)$
CDAN | $0.652(\pm 0.056)$ | $0.732(\pm 0.016)$ | $0.739(\pm 0.024)$ | ${\it 0.739(\pm 0.018)}$ | $0.728(\pm 0.029)$ | $0.697(\pm 0.031)$ | $0.642(\pm 0.065)$ | ${\bf 0.748(\pm 0.019)}$ | $0.713(\pm 0.045)$
DANN | $0.641(\pm 0.047)$ | $0.722(\pm 0.017)$ | ${\it 0.723(\pm 0.026)}$ | $0.721(\pm 0.025)$ | $0.714(\pm 0.024)$ | $0.687(\pm 0.034)$ | $0.644(\pm 0.046)$ | ${\bf 0.724(\pm 0.018)}$ | $0.710(\pm 0.035)$
DSAN | $0.653(\pm 0.060)$ | ${\it 0.748(\pm 0.008)}$ | $0.740(\pm 0.016)$ | $0.732(\pm 0.016)$ | $0.728(\pm 0.026)$ | $0.712(\pm 0.070)$ | $0.589(\pm 0.063)$ | ${\bf 0.757(\pm 0.016)}$ | $0.700(\pm 0.033)$
Avg. | $0.655(\pm 0.054)$ | ${\it 0.729(\pm 0.018)}$ | $0.729(\pm 0.024)$ | $0.725(\pm 0.023)$ | $0.717(\pm 0.028)$ | $0.700(\pm 0.052)$ | $0.660(\pm 0.057)$ | ${\bf 0.737(\pm 0.020)}$ | $0.712(\pm 0.045)$

| UCI-HAR
---|---
|  | Heuristic | Theoretical error guarantees |
Method | SO | TMV | TMR | TCR | SOR | IWV | DEV | IWA (ours) | TB
HoMM | $0.782(\pm 0.078)$ | ${\it 0.833(\pm 0.020)}$ | $0.818(\pm 0.023)$ | $0.818(\pm 0.022)$ | $0.783(\pm 0.040)$ | $0.809(\pm 0.095)$ | $0.800(\pm 0.098)$ | ${\bf 0.826(\pm 0.010)}$ | $0.854(\pm 0.039)$
AdvSKM | $0.724(\pm 0.059)$ | $0.791(\pm 0.024)$ | $0.800(\pm 0.022)$ | ${\it 0.810(\pm 0.022)}$ | $0.768(\pm 0.042)$ | $0.707(\pm 0.100)$ | $0.711(\pm 0.167)$ | ${\bf 0.800(\pm 0.022)}$ | $0.811(\pm 0.039)$
DIRT | $0.783(\pm 0.044)$ | ${\it 0.912(\pm 0.013)}$ | $0.907(\pm 0.009)$ | $0.890(\pm 0.016)$ | $0.756(\pm 0.036)$ | $0.807(\pm 0.107)$ | $0.808(\pm 0.112)$ | ${\bf 0.900(\pm 0.015)}$ | $0.928(\pm 0.034)$
DDC | $0.790(\pm 0.061)$ | $0.806(\pm 0.019)$ | $0.807(\pm 0.026)$ | ${\it 0.810(\pm 0.017)}$ | $0.756(\pm 0.108)$ | $0.724(\pm 0.066)$ | $0.734(\pm 0.109)$ | ${\bf 0.804(\pm 0.028)}$ | $0.792(\pm 0.013)$
CMD | $0.788(\pm 0.058)$ | ${\it 0.869(\pm 0.012)}$ | $0.849(\pm 0.014)$ | $0.839(\pm 0.023)$ | $0.731(\pm 0.066)$ | $0.804(\pm 0.064)$ | $0.812(\pm 0.080)$ | ${\bf 0.842(\pm 0.025)}$ | $0.888(\pm 0.037)$
MMDA | $0.785(\pm 0.018)$ | ${\it 0.819(\pm 0.022)}$ | $0.812(\pm 0.028)$ | $0.800(\pm 0.032)$ | $0.759(\pm 0.085)$ | $0.773(\pm 0.073)$ | $0.767(\pm 0.107)$ | ${\bf 0.807(\pm 0.025)}$ | $0.840(\pm 0.055)$
CoDATS | $0.760(\pm 0.037)$ | ${\it 0.854(\pm 0.022)}$ | $0.832(\pm 0.027)$ | $0.832(\pm 0.006)$ | $0.785(\pm 0.057)$ | $0.801(\pm 0.079)$ | $0.794(\pm 0.078)$ | ${\bf 0.846(\pm 0.016)}$ | $0.867(\pm 0.012)$
Deep-Coral | $0.790(\pm 0.035)$ | ${\it 0.810(\pm 0.007)}$ | $0.800(\pm 0.022)$ | $0.808(\pm 0.030)$ | $0.771(\pm 0.023)$ | $0.768(\pm 0.044)$ | $0.773(\pm 0.087)$ | ${\bf 0.808(\pm 0.016)}$ | $0.806(\pm 0.022)$
CDAN | $0.756(\pm 0.055)$ | $0.842(\pm 0.009)$ | ${\it 0.843(\pm 0.020)}$ | $0.840(\pm 0.034)$ | $0.802(\pm 0.080)$ | $0.781(\pm 0.072)$ | $0.687(\pm 0.068)$ | ${\bf 0.846(\pm 0.018)}$ | $0.853(\pm 0.026)$
DANN | $0.756(\pm 0.026)$ | ${\it 0.858(\pm 0.016)}$ | $0.856(\pm 0.033)$ | $0.856(\pm 0.033)$ | $0.800(\pm 0.057)$ | $0.763(\pm 0.032)$ | $0.780(\pm 0.043)$ | ${\bf 0.849(\pm 0.023)}$ | $0.847(\pm 0.007)$
DSAN | $0.762(\pm 0.032)$ | $0.849(\pm 0.023)$ | $0.843(\pm 0.033)$ | ${\it 0.854(\pm 0.025)}$ | $0.749(\pm 0.065)$ | $0.775(\pm 0.043)$ | $0.744(\pm 0.035)$ | ${\bf 0.858(\pm 0.023)}$ | $0.865(\pm 0.038)$
Avg. | $0.770(\pm 0.046)$ | ${\it 0.840(\pm 0.017)}$ | $0.833(\pm 0.023)$ | $0.832(\pm 0.024)$ | $0.769(\pm 0.060)$ | $0.774(\pm 0.070)$ | $0.765(\pm 0.090)$ | ${\bf 0.835(\pm 0.020)}$ | $0.850(\pm 0.029)$

| HHAR
---|---
|  | Heuristic | Theoretical error guarantees |
Method | SO | TMV | TMR | TCR | SOR | IWV | DEV | IWA (ours) | TB
HoMM | $0.739(\pm 0.044)$ | $0.757(\pm 0.014)$ | ${\it 0.759(\pm 0.013)}$ | $0.759(\pm 0.011)$ | $0.700(\pm 0.058)$ | $0.720(\pm 0.027)$ | $0.733(\pm 0.031)$ | ${\bf 0.759(\pm 0.007)}$ | $0.764(\pm 0.023)$
AdvSKM | $0.718(\pm 0.042)$ | ${\it 0.749(\pm 0.027)}$ | $0.742(\pm 0.032)$ | $0.748(\pm 0.034)$ | $0.676(\pm 0.046)$ | $0.730(\pm 0.051)$ | $0.728(\pm 0.051)$ | ${\bf 0.752(\pm 0.031)}$ | $0.749(\pm 0.025)$
DIRT | $0.728(\pm 0.026)$ | ${\it 0.803(\pm 0.011)}$ | $0.792(\pm 0.016)$ | $0.803(\pm 0.017)$ | $0.796(\pm 0.066)$ | $0.743(\pm 0.028)$ | $0.739(\pm 0.075)$ | ${\bf 0.816(\pm 0.008)}$ | $0.820(\pm 0.015)$
DDC | $0.716(\pm 0.063)$ | $0.748(\pm 0.014)$ | ${\it 0.750(\pm 0.009)}$ | $0.748(\pm 0.007)$ | $0.717(\pm 0.075)$ | $0.711(\pm 0.048)$ | $0.705(\pm 0.066)$ | ${\bf 0.748(\pm 0.012)}$ | $0.729(\pm 0.027)$
CMD | $0.748(\pm 0.027)$ | $0.760(\pm 0.014)$ | $0.764(\pm 0.006)$ | ${\it 0.767(\pm 0.007)}$ | $0.737(\pm 0.100)$ | ${\bf 0.775(\pm 0.031)}$ | $0.643(\pm 0.031)$ | $0.766(\pm 0.016)$ | $0.794(\pm 0.023)$
MMDA | $0.738(\pm 0.036)$ | ${\it 0.783(\pm 0.017)}$ | $0.781(\pm 0.016)$ | $0.780(\pm 0.015)$ | $0.698(\pm 0.038)$ | $0.719(\pm 0.036)$ | $0.731(\pm 0.047)$ | ${\bf 0.780(\pm 0.017)}$ | $0.785(\pm 0.035)$
CoDATS | $0.710(\pm 0.030)$ | $0.766(\pm 0.023)$ | $0.772(\pm 0.040)$ | ${\it 0.773(\pm 0.050)}$ | $0.722(\pm 0.064)$ | $0.739(\pm 0.028)$ | $0.739(\pm 0.040)$ | ${\bf 0.812(\pm 0.009)}$ | $0.785(\pm 0.039)$
Deep-Coral | $0.745(\pm 0.046)$ | ${\it 0.766(\pm 0.012)}$ | $0.762(\pm 0.015)$ | $0.766(\pm 0.027)$ | $0.681(\pm 0.073)$ | $0.754(\pm 0.054)$ | $0.758(\pm 0.244)$ | ${\bf 0.764(\pm 0.006)}$ | $0.776(\pm 0.023)$
CDAN | $0.728(\pm 0.039)$ | $0.762(\pm 0.012)$ | $0.758(\pm 0.017)$ | $0.764(\pm 0.016)$ | ${\it 0.765(\pm 0.063)}$ | $0.774(\pm 0.035)$ | $0.775(\pm 0.036)$ | ${\bf 0.816(\pm 0.011)}$ | $0.790(\pm 0.038)$
DANN | $0.757(\pm 0.057)$ | ${\it 0.779(\pm 0.012)}$ | $0.774(\pm 0.009)$ | $0.773(\pm 0.011)$ | $0.722(\pm 0.103)$ | $0.798(\pm 0.041)$ | $0.793(\pm 0.045)$ | ${\bf 0.818(\pm 0.009)}$ | $0.807(\pm 0.020)$
DSAN | $0.721(\pm 0.053)$ | ${\it 0.803(\pm 0.010)}$ | $0.797(\pm 0.014)$ | $0.802(\pm 0.007)$ | $0.724(\pm 0.065)$ | $0.741(\pm 0.033)$ | $0.596(\pm 0.031)$ | ${\bf 0.825(\pm 0.008)}$ | $0.826(\pm 0.046)$
Avg. | $0.732(\pm 0.042)$ | $0.771(\pm 0.015)$ | $0.768(\pm 0.017)$ | ${\it 0.771(\pm 0.018)}$ | $0.722(\pm 0.068)$ | $0.746(\pm 0.037)$ | $0.722(\pm 0.063)$ | ${\bf 0.787(\pm 0.012)}$ | $0.784(\pm 0.028)$

| WISDM
---|---
|  | Heuristic | Theoretical error guarantees |
Method | SO | TMV | TMR | TCR | SOR | IWV | DEV | IWA (ours) | TB
HoMM | $0.753(\pm 0.054)$ | $0.741(\pm 0.026)$ | $0.738(\pm 0.031)$ | $0.739(\pm 0.047)$ | ${\it 0.775(\pm 0.062)}$ | ${\bf 0.753(\pm 0.054)}$ | $0.740(\pm 0.054)$ | $0.728(\pm 0.021)$ | $0.774(\pm 0.037)$
AdvSKM | $0.747(\pm 0.050)$ | $0.771(\pm 0.043)$ | ${\it 0.781(\pm 0.055)}$ | $0.779(\pm 0.035)$ | $0.742(\pm 0.062)$ | $0.747(\pm 0.050)$ | $0.747(\pm 0.135)$ | ${\bf 0.777(\pm 0.031)}$ | $0.779(\pm 0.041)$
DIRT | $0.738(\pm 0.038)$ | $0.792(\pm 0.015)$ | ${\it 0.797(\pm 0.024)}$ | $0.797(\pm 0.037)$ | $0.756(\pm 0.071)$ | $0.738(\pm 0.059)$ | $0.797(\pm 0.059)$ | ${\bf 0.798(\pm 0.018)}$ | $0.816(\pm 0.063)$
DDC | $0.741(\pm 0.071)$ | $0.780(\pm 0.032)$ | $0.779(\pm 0.052)$ | ${\it 0.787(\pm 0.049)}$ | $0.737(\pm 0.071)$ | $0.741(\pm 0.076)$ | $0.741(\pm 0.063)$ | ${\bf 0.782(\pm 0.038)}$ | $0.770(\pm 0.060)$
CMD | $0.710(\pm 0.088)$ | ${\it 0.772(\pm 0.021)}$ | $0.765(\pm 0.032)$ | $0.767(\pm 0.040)$ | $0.728(\pm 0.092)$ | $0.713(\pm 0.084)$ | $0.686(\pm 0.113)$ | ${\bf 0.773(\pm 0.032)}$ | $0.742(\pm 0.071)$
MMDA | $0.759(\pm 0.047)$ | ${\it 0.789(\pm 0.017)}$ | $0.772(\pm 0.030)$ | $0.745(\pm 0.035)$ | $0.754(\pm 0.050)$ | $0.759(\pm 0.047)$ | $0.750(\pm 0.047)$ | ${\bf 0.790(\pm 0.018)}$ | $0.775(\pm 0.030)$
CoDATS | $0.711(\pm 0.039)$ | ${\it 0.775(\pm 0.018)}$ | $0.757(\pm 0.027)$ | $0.751(\pm 0.020)$ | $0.682(\pm 0.057)$ | $0.709(\pm 0.039)$ | $0.735(\pm 0.054)$ | ${\bf 0.764(\pm 0.015)}$ | $0.770(\pm 0.019)$
Deep-Coral | $0.694(\pm 0.030)$ | $0.717(\pm 0.041)$ | ${\it 0.723(\pm 0.037)}$ | $0.713(\pm 0.035)$ | $0.664(\pm 0.055)$ | $0.694(\pm 0.030)$ | $0.670(\pm 0.149)$ | ${\bf 0.723(\pm 0.026)}$ | $0.736(\pm 0.044)$
CDAN | $0.760(\pm 0.057)$ | $0.762(\pm 0.048)$ | $0.762(\pm 0.046)$ | ${\it 0.781(\pm 0.051)}$ | $0.768(\pm 0.074)$ | $0.760(\pm 0.057)$ | $0.750(\pm 0.091)$ | ${\bf 0.765(\pm 0.040)}$ | $0.779(\pm 0.049)$
DANN | $0.724(\pm 0.042)$ | $0.789(\pm 0.018)$ | ${\it 0.802(\pm 0.028)}$ | $0.796(\pm 0.036)$ | $0.745(\pm 0.026)$ | $0.720(\pm 0.042)$ | $0.702(\pm 0.053)$ | ${\bf 0.778(\pm 0.019)}$ | $0.765(\pm 0.044)$
DSAN | $0.759(\pm 0.030)$ | $0.765(\pm 0.024)$ | ${\it 0.769(\pm 0.034)}$ | $0.756(\pm 0.025)$ | $0.757(\pm 0.068)$ | ${\bf 0.759(\pm 0.030)}$ | $0.663(\pm 0.025)$ | $0.722(\pm 0.013)$ | $0.779(\pm 0.044)$
Avg. | $0.736(\pm 0.050)$ | ${\it 0.768(\pm 0.027)}$ | $0.768(\pm 0.036)$ | $0.765(\pm 0.037)$ | $0.737(\pm 0.062)$ | $0.736(\pm 0.052)$ | $0.726(\pm 0.077)$ | ${\bf 0.764(\pm 0.025)}$ | $0.771(\pm 0.046)$

### Acknowledgments

The ELLIS Unit Linz, the LIT AI Lab, and the Institute for Machine Learning
are supported by the Federal State Upper Austria. IARAI is supported by Here
Technologies. We thank the projects AI-MOTION (LIT-2018-6-YOU-212), AI-SNN
(LIT-2018-6-YOU-214), DeepFlood (LIT-2019-8-YOU-213), Medical Cognitive
Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI
(FFG-872172), DL for GranularFlow (FFG-871302), AIRI FG 9-N (FWF-36284,
FWF-36235), and ELISE (H2020-ICT-2019-3 ID: 951847). We further thank Audi.JKU
Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL),
FILL GmbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB
Biopharma SRL, Merck Healthcare KGaA, Verbund AG, TÜV Austria, Frauscher
Sensonic, and the NVIDIA Corporation. The research reported in this paper has
been funded by the Federal Ministry for Climate Action, Environment, Energy,
Mobility, Innovation and Technology (BMK), the Federal Ministry for Digital
and Economic Affairs (BMDW), and the Province of Upper Austria in the frame of
the COMET–Competence Centers for Excellent Technologies Programme and the
COMET Module S3AI managed by the Austrian Research Promotion Agency FFG.

## References

  * Abadi et al. (2015) M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  Software available from tensorflow.org.
  * Ahmed et al. (2022) W. Ahmed, P. Morerio, and V. Murino.  Cleaning noisy labels by negative ensemble learning for source-free unsupervised domain adaptation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 1616–1625, 2022.
  * Al-Stouhi & Reddy (2011) S. Al-Stouhi and C. K. Reddy.  Adaptive boosting for transfer learning using dynamic updates.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pp. 60–75. Springer, 2011.
  * Anguita et al. (2013) D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz.  A public domain dataset for human activity recognition using smartphones.  _European Symposium on Artificial Neural Networks_ , pp. 437–442, 2013.
  * Athiwaratkun et al. (2019) B. Athiwaratkun, M. Finzi, P. Izmailov, and A. G. Wilson.  There are many consistent explanations of unlabeled data: Why you should average.  _International Conference on Learning Representations (2019)_ , 2019\.
  * Ben-David et al. (2007) S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira.  Analysis of representations for domain adaptation.  In _Advances in Neural Information Processing Systems_ , pp. 137–144, 2007.
  * Ben-David et al. (2010) S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan.  A theory of learning from different domains.  _Machine Learning_ , 79(1-2):151–175, 2010.
  * Bickel et al. (2007) S. Bickel, M. Brückner, and T. Scheffer.  Discriminative learning for differing training and test distributions.  In _Proceedings of the 24th international conference on Machine learning_ , pp. 81–88, 2007.
  * Bietti & Mairal (2017) A. Bietti and J. Mairal.  Invariance and stability of deep convolutional representations.  _Advances in neural information processing systems_ , 30, 2017.
  * Bietti & Mairal (2019) A. Bietti and J. Mairal.  Group invariance, stability to deformations, and complexity of deep convolutional representations.  _The Journal of Machine Learning Research_ , 20(1):876–924, 2019.
  * Biewald (2020) L. Biewald.  Experiment tracking with weights and biases, 2020.  URL https://www.wandb.com/.  Software available from wandb.com.
  * Blitzer et al. (2006) J. Blitzer, R. McDonald, and F. Pereira.  Domain adaptation with structural correspondence learning.  In _Proceedings of the 2006 conference on empirical methods in natural language processing_ , pp. 120–128, 2006.
  * Breiman (1994) L. Breiman.  Bagging predictors.  Technical Report 421, Department of Statistics, UC Berkeley, 1994.
  * Breiman (1996a) L. Breiman.  Bagging predictors.  _Machine Learning_ , 26(2):123–140, 1996a.
  * Breiman (1996b) L. Breiman.  Stacked regressions.  _Machine Learning_ , 24(1):49–64, 1996b.
  * Breiman (1998) L. Breiman.  Arcing classifier (with discussion and a rejoinder by the author).  _The Annals of Statistics_ , 26(3):801–849, 1998\.  doi: 10.1214/aos/1024691079.
  * Caponnetto & De Vito (2005) A. Caponnetto and E. De Vito.  Risk bounds for regularized least-squares algorithm with operatorvalued kernels.  Technical report, CBCL paper 249/CSAIL-TR-2005-031, MIT, 2005.
  * Caponnetto & De Vito (2007) A. Caponnetto and E. De Vito.  Optimal rates for the regularized least-squares algorithm.  _Foundations of Computational Mathematics_ , 7(3):331–368, 2007.
  * Chen et al. (2020) C. Chen, Z. Fu, Z. Chen, S. Jin, Z. Cheng, X. Jin, and X.-S. Hua.  Homm: Higher-order moment matching for unsupervised domain adaptation.  _Association for the Advancement of Artificial Intelligence (AAAI)_ , 2020.
  * Chen et al. (2012) M. Chen, Z. Xu, K. Weinberger, and F. Sha.  Marginalized denoising autoencoders for domain adaptation.  _Proceedings of the International Conference on Machine Learning_ , pp. 767–774, 2012.
  * Cucker & Smale (2002) F. Cucker and S. Smale.  On the mathematical foundations of learning.  _Bulletin of the American mathematical society_ , 39(1):1–49, 2002.
  * da Costa-Luis (2019) C. O. da Costa-Luis.  Tqdm: A fast, extensible progress meter for python and cli.  _Journal of Open Source Software_ , 4(37):1277, 2019.
  * Dai et al. (2007) W. Dai, Q. Yang, G. R. Xue, and Y. Yu.  Boosting for transfer learning.  In _Proceedings of the 24th International Conference on Machine Learning_ , pp. 193–200, 2007.
  * Dong et al. (2020) X. Dong, Z. Yu, W. Cao, Y. Shi, and Q. Ma.  A survey on ensemble learning.  _Frontiers of Computer Science_ , 14(2):241–258, 2020.
  * Duan et al. (2012) L. Duan, I. W. Tsang, and D. Xu.  Domain transfer multiple kernel learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 34(3):465–479, 2012.
  * Dudley (2002) R. M. Dudley.  _Real analysis and probability_ , volume 74.  Cambridge University Press, 2002.
  * Eldele et al. (2021) E. Eldele, Z. Chen, C. Liu, M. Wu, C.-K. Kwoh, X. Li, and C. Guan.  An attention-based deep learning approach for sleep stage classification with single-channel eeg.  _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , 2021.
  * Fermanian et al. (2021) A. Fermanian, P. Marion, J. P. Vert, and G. Biau.  Framing rnn as a kernel method: A neural ode approach.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * French et al. (2018) G. French, M. Mackiewicz, and M. Fisher.  Self-ensembling for visual domain adaptation.  _International Conference on Learning Representations_ , 2018.
  * Ganin et al. (2016) Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky.  Domain-adversarial training of neural networks.  _Journal of Machine Learning Research_ , 17(Jan):1–35, 2016.
  * Gizewski et al. (2022) E. R. Gizewski, L. Mayer, B. A. Moser, D. H. Nguyen, S. Pereverzyev Jr, S. V. Pereverzyev, N. Shepeleva, and W. Zellinger.  On a regularization of unsupervised domain adaptation in RKHS.  _Applied and Computational Harmonic Analysis_ , 57:201–227, 2022.
  * Goldberger et al. (2000) A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley.  Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals.  _Circulation_ , 101(23):215–220, 2000.
  * Goodfellow et al. (2016) I. Goodfellow, Y. Bengio, and A. Courville.  _Deep learning_.  MIT press, 2016.
  * Gretton et al. (2006) A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, and A. J. Smola.  A kernel method for the two-sample-problem.  In _Advances in Neural Information Processing Systems_ , pp. 513–520, 2006.
  * Ha & Youn (2021) J. M. Ha and B. D. Youn.  A health data map-based ensemble of deep domain adaptation under inhomogeneous operating conditions for fault diagnosis of a planetary gearbox.  _IEEE Access_ , 9:79118–79127, 2021.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hochreiter & Schmidhuber (1994) S. Hochreiter and J. Schmidhuber.  Simplifying neural nets by discovering flat minima.  _Advances in neural information processing systems_ , 7, 1994.
  * Hochreiter & Schmidhuber (1997) S. Hochreiter and J. Schmidhuber.  Flat minima.  _Neural computation_ , 9(1):1–42, 1997.
  * Hoffman et al. (2018) J. Hoffman, M. Mohri, and N. Zhang.  Algorithms and theory for multiple-source adaptation.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Huang et al. (2006) J. Huang, A. Gretton, K. Borgwardt, B. Schölkopf, and A. Smola.  Correcting sample selection bias by unlabeled data.  _Advances in neural information processing systems_ , 19, 2006.
  * III & Marcu (2006) H. Daume III and D. Marcu.  Domain adaptation for statistical classifiers.  _Journal of artificial Intelligence research_ , 26:101–126, 2006.
  * Jiao et al. (2019) J. Jiao, M. Zhao, J. Lin, and C. Ding.  Classifier inconsistency-based domain adaptation network for partial transfer intelligent diagnosis.  _IEEE Transactions on Industrial Informatics_ , 16(9):5965–5974, 2019.
  * Kanamori et al. (2009) T. Kanamori, S. Hido, and M. Sugiyama.  A least-squares approach to direct importance estimation.  _The Journal of Machine Learning Research_ , 10:1391–1445, 2009.
  * Kanamori et al. (2012) T. Kanamori, T. Suzuki, and M. Sugiyama.  Statistical analysis of kernel-based least-squares density-ratio estimation.  _Machine Learning_ , 86(3):335–367, 2012.
  * Kang et al. (2020) G. Kang, L. Jiang, Y. Wei, Y. Yang, and A. G. Hauptmann.  Contrastive adaptation network for single-and multi-source domain adaptation.  _IEEE transactions on pattern analysis and machine intelligence_ , 2020.
  * Kingma & Ba (2014) D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kouw et al. (2019) W. M. Kouw, J. H. Krijthe, and M. Loog.  Robust importance-weighted cross-validation under sample selection bias.  In _IEEE International Workshop on Machine Learning for Signal Processing_ , pp. 1–6. IEEE, 2019.
  * Krizhevsky et al. (2012) A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_ , pp. 1097–1105, 2012.
  * Kwapisz et al. (2011) J. R. Kwapisz, G. M. Weiss, and S. A. Moore.  Activity recognition using cell phone accelerometers.  _Sigkdd Explorations_ , 12(2):74–82, 2011.
  * Laine & Aila (2017) S. Laine and T. Aila.  Temporal ensembling for semi-supervised learning.  _International Conference on Learning Representations (ICLR)_ , 2017\.
  * Liu & Xue (2021) Q. Liu and H. Xue.  Adversarial spectral kernel matching for unsupervised time series domain adaptation.  _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ , 30, 2021.
  * Long et al. (2015) M. Long, Y. Cao, J. Wang, and M. Jordan.  Learning transferable features with deep adaptation networks.  In _Proceedings of the International Conference on Machine Learning_ , pp. 97–105, 2015.
  * Long et al. (2018) M. Long, Z. Cao, J. Wang, and M. I. Jordan.  Conditional adversarial domain adaptation.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 31, 2018.
  * Louizos et al. (2016) C. Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel.  The variational fair auto encoder.  _International Conference on Learning Representations_ , 2016.
  * Ma & Wu (2022) C. Ma and L. Wu.  The barron space and the flow-induced function spaces for neural network models.  _Constructive Approximation_ , 55(1):369–406, 2022.
  * Mayr et al. (2016) A. Mayr, G. Klambauer, T. Unterthiner, and S. Hochreiter.  Deeptox: toxicity prediction using deep learning.  _Frontiers in Environmental Science_ , 3:80, 2016.
  * Musgrave et al. (2021) K. Musgrave, S. Belongie, and S.-N. Lim.  Unsupervised domain adaptation: A reality check.  _arXiv preprint arXiv:2111.15672_ , 2021.
  * Nozza et al. (2016) D. Nozza, E. Fersini, and E. Messina.  Deep learning and ensemble methods for domain adaptation.  In _IEEE 28th International Conference on Tools with Artificial Intelligence (ICTAI)_ , pp. 184–189. IEEE, 2016.
  * Pan & Yang (2010) S. J. Pan and Q. Yang.  A survey on transfer learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 22(10):1345–1359, 2010.
  * Paszke et al. (2017) A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  2017\.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al.  Scikit-learn: Machine learning in python.  _the Journal of machine Learning research_ , 12:2825–2830, 2011.
  * Peng et al. (2018) X. Peng, B. Usmanand N. Kaushikand D. Wang, J. Hoffman, and K. Saenko.  Visda: A synthetic-to-real benchmark for visual domain adaptation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 2021–2026, 2018.
  * Peng et al. (2019) X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 1406–1415, 2019.
  * Pereverzyev (2022) S. V. Pereverzyev.  _An Introduction to Artificial Intelligence based on Reproducing Kernel Hilbert Spaces_.  Birkhäuser Cham, 2022.
  * Pinelis (1992) I. Pinelis.  An approach to inequalities for the distributions of infinite-dimensional martingales.  In _Probability in Banach Spaces, 8: Proceedings of the Eighth International Conference_ , pp. 128–134. Springer, 1992.
  * Ragab et al. (2022) M. Ragab, E. Eldele, W. L. Tan, C.-S. Foo, Z. Chen, M. Wu, C.-K. Kwoh, and X. Li.  Adatime: A benchmarking suite for domain adaptation on time series data.  _arXiv preprint arXiv:2203.08321_ , 2022.
  * Rahman et al. (2020) M. M. Rahman, C. Fookes, M. Baktashmotlagh, and S. Sridharan.  On minimum discrepancy estimation for deep domain adaptation.  _Domain Adaptation for Visual Understanding_ , 2020.
  * Rakshit et al. (2019) S. Rakshit, B. Banerjee, G. Roig, and S. Chaudhuri.  Unsupervised multi-source domain adaptation driven by deep adversarial ensemble learning.  In _German Conference on Pattern Recognition_ , pp. 485–498. Springer, 2019.
  * Razar & Samothrakis (2019) H. Razar and S. Samothrakis.  Bagging adversarial neural networks for domain adaptation in non-stationary eeg.  In _2019 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–7. IEEE, 2019.
  * Rosasco et al. (2010) L. Rosasco, M. Belkin, and E. De Vito.  On learning with integral operators.  _Journal of Machine Learning Research_ , 11(2), 2010.
  * Saenko et al. (2010) K. Saenko, B. Kulis, M. Fritz, and T. Darrell.  Adapting visual category models to new domains.  In _European conference on computer vision_ , pp. 213–226. Springer, 2010.
  * Saito et al. (2017) K. Saito, Y. Ushiku, and T. Harada.  Asymmetric tri-training for unsupervised domain adaptation.  In _International Conference on Machine Learning_ , pp. 2988–2997. PMLR, 2017.
  * Saito et al. (2021) K. Saito, D. Kim, P. Teterwak, S. Sclaroff, T. Darrell, and K. Saenko.  Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9184–9193, 2021.
  * Schapire (1990) R. E. Schapire.  The strength of weak learnability.  _Machine Learning_ , 5:197–227, 1990.
  * Shimodaira (2000) H. Shimodaira.  Improving predictive inference under covariate shift by weighting the log-likelihood function.  _Journal of Statistical Planning and Inference_ , 90(2):227–244, 2000.
  * Shorten & Khoshgoftaar (2019) C. Shorten and T. M. Khoshgoftaar.  A survey on image data augmentation for deep learning.  _Journal of Big Data_ , 6(1):1–48, 2019.
  * Shu et al. (2018) R. Shu, H. Bui, H. Narui, and S. Ermon.  A dirt-t approach to unsupervised domain adaptation.  _International Conference on Learning Representations (ICLR)_ , 2018\.
  * Stisen et al. (2015) A. Stisen, H. Blunck, S. Bhattacharya, T. S. Prentow, M. B. Kjærgaard, A. Dey, T. Sonne, and M. M. Jensen.  Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition.  In _Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems_ , SenSys ’15, pp. 127–140, New York, NY, USA, 2015. Association for Computing Machinery.  ISBN 9781450336314.  doi: 10.1145/2809695.2809718.
  * Strang (1980) G. Strang.  _Linear algebra and its applications._ Orlando, FL, Academic Press, Inc., 1980.
  * Sugiyama et al. (2007) M. Sugiyama, M. Krauledat, and K. M. Müller.  Covariate shift adaptation by importance weighted cross validation.  _Journal of Machine Learning Research_ , 8(5), 2007.
  * Sugiyama et al. (2012) M. Sugiyama, T. Suzuki, and T. Kanamori.  _Density ratio estimation in machine learning_.  Cambridge University Press, 2012.
  * Sun & Saenko (2016) B. Sun and K. Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In _Proceedings of the European Conference on Computer Vision_ , pp. 443–450, 2016.
  * Sun et al. (2017) B. Sun, J. Feng, and K. Saenko.  Correlation alignment for unsupervised domain adaptation.  _Domain Adaptation in Computer Vision Applications_ , pp. 153–171, 2017.
  * Sun (2012) W. Tuand S. Sun.  Dynamical ensemble learning with model-friendly classifiers for domain adaptation.  In _Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012)_ , pp. 1181–1184. IEEE, 2012.
  * Tarvainen & Valpola (2017) A. Tarvainen and H. Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  _Advances in neural information processing systems_ , 30, 2017.
  * Teschl (2022a) G. Teschl.  Topics in Linear and Nonlinear Functional Analysis.  _Amer. Math. Soc., Providence, to appear_ , 2022a.
  * Teschl (2022b) G. Teschl.  Topics in Real Analysis.  _Amer. Math. Soc., Providence, to appear_ , 2022b.
  * Tzeng et al. (2014) E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell.  Deep domain confusion: Maximizing for domain invariance.  _arXiv preprint arXiv:1412.3474_ , 2014.
  * Varsavsky et al. (2020) T. Varsavsky, M. Orbes-Arteaga, C. H. Sudre, M. S. Graham, P. Nachev, and M. J. Cardoso.  Test-time unsupervised domain adaptation.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pp. 428–436. Springer, 2020.
  * Wilson & Cook (2020) G. Wilson and D. J. Cook.  A survey of unsupervised deep domain adaptation.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 11(5):1–46, 2020.
  * Wilson et al. (2020) G. Wilson, J. R. Doppa, and D. J. Cook.  Multi-source deep domain adaptation with weak supervision for time-series sensor data.  _Special Interest Group on Knowledge Discovery and Data Mining (SIGKDD)_ , 2020.
  * Wolpert (1992) D. H. Wolpert.  Stacked generalization.  _Neural Networks_ , 5:214–259, 1992.
  * Wortsman et al. (2022) M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  _arXiv preprint arXiv:2203.05482_ , 2022.
  * Xia et al. (2013) R. Xia, C. Zong, X. Hu, and E. Cambria.  Feature ensemble plus sample selection: domain adaptation for sentiment classification.  _IEEE Intelligent Systems_ , 28(3):10–18, 2013\.
  * Xu et al. (2018) R. Xu, Z. Chen, W. Zuo, J. Yan, and L. Lin.  Deep cocktail network: Multi-source unsupervised domain adaptation with category shift.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3964–3973, 2018.
  * Yang et al. (2012) J. B. Yang, Q. Mao, Q. L. Xiang, I. W.-H. Tsangand K. M. A. Chai, and H. L. Chieu.  Domain adaptation for coreference resolution: An adaptive ensemble approach.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pp. 744–753, 2012.
  * You et al. (2019) K. You, X. Wang, M. Long, and M. Jordan.  Towards accurate model selection in deep unsupervised domain adaptation.  In _Proceedings of the International Conference on Machine Learning_ , pp. 7124–7133. PMLR, 2019.
  * Zellinger et al. (2017) W. Zellinger, T. Grubinger, E. Lughofer, T. Natschläger, and S. Saminger-Platz.  Central moment discrepancy (cmd) for domain-invariant representation learning.  _International Conference on Learning Representations_ , 2017.
  * Zellinger et al. (2020) W. Zellinger, T. Grubinger, M. Zwick, E. Lughofer, H. Schöner, T. Natschläger, and S. Saminger-Platz.  Multi-source transfer learning of time series in cyclical manufacturing.  _Journal of Intelligent Manufacturing_ , 31(3):777–787, 2020.
  * Zellinger et al. (2021) W. Zellinger, N. Shepeleva, M.-C. Dinu, H. Eghbal-zadeh, H. Nguyen, B. Nessler, S. Pereverzyev, and B. A. Moser.  The balancing principle for parameter choice in distance-regularized domain adaptation.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Zhang et al. (2015) K. Zhang, M. Gong, and B. Schölkopf.  Multi-source domain adaptation: A causal view.  In _Twenty-ninth AAAI conference on artificial intelligence_ , 2015\.
  * Zhou et al. (2021) K. Zhou, Y. Yang, Y. Qiao, and T. Xiang.  Domain adaptive ensemble learning.  _IEEE Transactions on Image Processing_ , 30:8008–8018, 2021.
  * Zhu et al. (2021) Y. Zhu, F. Zhuang, J. Wang, G. Ke, J. Chen, J. Bian, H. Xiong, and Q. He.  Deep subdomain adaptation network for image classification.  _IEEE Transactions on Neural Networks and Learning Systems_ , 32(4):1713–1722, 2021.
  * Zou et al. (2020) D. Zou, Q. Zhu, and P. Yan.  Unsupervised domain adaptation with dual-scheme fusion network for medical image segmentation.  In _IJCAI_ , pp. 3291–3298, 2020.
  * Zou et al. (2018) Y. Zou, Z. Yu, B. V. K. Kumar, and J. Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 289–305, 2018.

##  Appendix A Notation and Proof of Main Result

The aim of this section is to give a full proof of our main result, Theorem 1
in the main paper. We start by introducing and summarizing the notation and
the required concepts from functional analysis and measure theory, so that we
can state and prove the required lemmas.

##### Summary of Notation

  * •

Spaces: input space $\mathcal{X}\subset\mathbb{R}^{d_{1}}$ and label space
$\mathcal{Y}$ with inner product $\left\langle.,.\right\rangle_{\mathcal{Y}}$.
$\mathcal{Y}$ is assumed to be a separable Hilbert space such that for the
associated norm $\left\lVert y\right\rVert_{\mathcal{Y}}\leq y_{0}$ holds for
all $y\in\mathcal{Y}$ and some $y_{0}>0$. Note that this setting is more
general than the one from the main text, where we assumed
$\mathcal{Y}\subset\mathbb{R}^{d_{2}}$ (the simplification in the main text
improves readability and respectation of space limits).

  * •

Datasets and Distributions: Source data set:
$(\mathbf{x},{\mathbf{y}})=((x_{1},y_{1}),\ldots,(x_{n},y_{n}))\in\left(\mathcal{X}\times\mathcal{Y}\right)^{n}$
independently drawn according to source distribution $p$ on
$\mathcal{X}\times\mathcal{Y}$ and an unlabeled target dataset
$\mathbf{x}^{\prime}=(x_{1}^{\prime},\ldots,x_{m}^{\prime})\in\mathcal{X}^{m}$
independently drawn according marginal distribution $q_{\mathcal{X}}$ of
target distribution $q$ on $\mathcal{X}\times\mathcal{Y}$ (the corresponding
marginal distribution of $p$ on $\mathcal{X}$ is similarly denoted as
$p_{\mathcal{X}}$).

  * •

Source Risk:
$\mathcal{R}_{p}(f)=\int_{\mathcal{X}\times\mathcal{Y}}\left\lVert
f(x)-y\right\rVert_{\mathcal{Y}}^{2}\mathop{}\\!\mathrm{d}p(x,y)$.

  * •

Source Regression function
$f_{p}(x)=\int_{\mathcal{Y}}y\mathop{}\\!\mathrm{d}p(y|x)$. (Vector valued)
integral in the sense of Lebesgue-Bochner.

  * •

Target Risk:
$\mathcal{R}_{q}(f)=\int_{\mathcal{X}\times\mathcal{Y}}\left\lVert
f(x)-y\right\rVert_{\mathcal{Y}}^{2}\mathop{}\\!\mathrm{d}q(x,y)$

  * •

Target Regression function
$f_{q}(x)=\int_{\mathcal{Y}}y\mathop{}\\!\mathrm{d}q(y|x)$. (Vector valued)
integral in the sense of Lebesgue-Bochner.

##### Problem

  * •

Given: sequence $f_{1},\ldots,f_{l}:\mathcal{X}\to\mathcal{Y}$ of models,
source sample $(\mathbf{x},{\mathbf{y}})$ and unlabeled target sample
$\mathbf{x}^{\prime}$

  * •

Aim: find aggretation $f=\sum_{i=1}^{l}c_{i}f_{i}$ with minimal
$\mathcal{R}_{q}(f)$.

##### Main Assumptions

  * •

covariate shift: $p(y|x)=q(y|x)$ and thus $f_{p}=f_{q}$.

  * •

bounded density ratio: there is $\beta:\mathcal{X}\to[0,B]$ such that
$\mathop{}\\!\mathrm{d}q_{\mathcal{X}}(x)=\beta(x)\mathop{}\\!\mathrm{d}p_{\mathcal{X}}(x)$.

Existence of the associated conditional probability measures is guaranteed by
the fact that $\mathcal{X}\times\mathcal{Y}$ is Polish (a separable and
complete metric space), c.f. Dudley (2002, Theorem 10.2.2.).

##### Notation from functional analysis/operator theory

Let $\mathbf{U}$ and $\mathbf{V}$ denote separable Hilbert spaces (i.e. they
admit countable orthonormal bases) with associated inner products
$\left\langle.,.\right\rangle_{\mathbf{U}}$ (or
$\left\langle.,.\right\rangle_{\mathbf{V}}$, respectively). Let us briefly
recall some notions from functional analysis that we need in order to set up
our theory. There are lots of standard references on these aspects, e.g.
Teschl (2022a) and Teschl (2022b):

  * •

$\mathcal{L}(\mathbf{U},\mathbf{V})$: space of bounded linear operators
$\mathbf{U}\to\mathbf{V}$ with uniform norm
$\left\lVert.\right\rVert_{\mathcal{L}(\mathbf{U},\mathbf{V})}$.
$\mathcal{L}(\mathbf{U})$: space of bounded linear operators
$\mathbf{U}\to\mathbf{U}$.

  * •

For $A\in\mathcal{L}(\mathbf{U},\mathbf{V})$, its adjoint is denoted by
$A^{*}\in\mathcal{L}(\mathbf{V},\mathbf{U})$ (and uniquely defined by the
equation $\left\langle Au,v\right\rangle_{\mathbf{V}}=\left\langle
u,A^{*}v\right\rangle_{\mathbf{U}}$ for any $u\in\mathbf{U}$,
$v\in\mathbf{V}$).

  * •

If $A\in\mathcal{L}(\mathbf{U})$ and $A=A^{*}$: $A$ is called self-adjoint.

  * •

If $A\in\mathcal{L}(\mathbf{U})$ is self adjoint and $\left\langle
Au,u\right\rangle_{\mathbf{U}}\geq 0$ for any $u\in\mathbf{U}$, then $A$ is
called positive. Equivalently: there exists (unique) bounded and self-adjoint
$B:=\sqrt{A}\in\mathcal{L}(\mathbf{U})$ such that $B^{2}=A$.

  * •

Trace of an operator $A\in\mathcal{L}(\mathbf{U})$:
$Tr(A)=\sum_{k}\left\langle Ae_{k},e_{k}\right\rangle_{\mathbf{U}}$ for any
orthonormal basis $\left(e_{k}\right)_{k=1}^{\infty}$ of $\mathbf{U}$
(independent of choice of basis). If $Tr(A)<\infty$: $A$ is called trace
class.

  * •

$\mathcal{L}_{2}(\mathbf{U})$: separable Hilbert space of Hilbert-Schmidt
operators on $\mathbf{U}$ with scalar product $\left\langle
A,B\right\rangle_{\mathcal{L}_{2}(\mathbf{U})}=Tr(B^{*}A)$ and norm
$\left\lVert
A\right\rVert_{\mathcal{L}_{2}(\mathbf{U})}=\sqrt{Tr(A^{*}A)}\geq\left\lVert
A\right\rVert_{\mathcal{L}(\mathbf{U})}.$

  * •

$A:\mathbf{U}\to\mathbf{V}$ is called Hilbert-Schmidt, if $A^{*}A$ is trace
class. Also here: $\left\lVert
A\right\rVert_{\mathcal{L}(\mathbf{U},\mathbf{V})}\leq\sqrt{Tr(A^{*}A)}$

  * •

For (probability) measure $q$ on $\mathcal{X}$ (or $\mathcal{Y}$) and
appropriate functions $F:\mathcal{X}\to\mathbf{U}$ (e.g. strongly measurable
and $\left\lVert F\right\rVert_{\mathbf{U}}$ is integrable wrt. $q$) we denote
the usual ($\mathbf{U}$-valued) Bochner integral of $F$ as
$\int_{\mathcal{X}}F(x)\mathop{}\\!\mathrm{d}q(x)$. We denote the associated
$L^{p}$-spaces by $L^{p}(\mathcal{X},q,\mathbf{U})$, or $L^{p}(q)$ for short,
if the associated spaces are clear from the context.

##### Assumptions on models

We assume that the regression function $f^{*}=f_{p}=f_{q}$ as well as the
models $f_{1},...,f_{l}$ belong to a hypothesis space $\mathcal{H}\subseteq
C(\mathcal{X},\mathcal{Y})\subseteq L^{2}(p_{\mathcal{X}})\cap
L^{2}(q_{\mathcal{X}})$, where $C(\mathcal{X},\mathcal{Y})$ denotes the space
of bounded continous functions $\mathcal{X}\to\mathcal{Y}$. The space
$\mathcal{H}$ should satisfy the following assumptions, which are discussed in
much greater detail in Caponnetto & De Vito (2007) and Caponnetto & De Vito
(2005):

######  Hypothesis 1.

(Caponnetto & De Vito, 2007) The space $\mathcal{H}$ is a separable Hilbert
space of functions $f:\mathcal{X}\rightarrow\mathcal{Y}$ such that:

  * •

For all $x\in\mathcal{X}$ there is a Hilbert-Schmidt operator
$K_{x}:\mathcal{Y}\rightarrow\mathcal{H}$ satisfying

$\displaystyle f(x)=K_{x}^{*}f,\hskip 8.5359ptf\in\mathcal{H},$ (5)

  * •

The function from $\mathcal{X}\times\mathcal{X}$ to $\mathbb{R}$

$\displaystyle(x,t)\mapsto\left\langle
K_{t}v,K_{x}w\right\rangle_{\mathcal{H}}\text{ is measurable }\forall
v,w\in\mathcal{Y};$ (6)

  * •

There is $\kappa>0$ such that

$\displaystyle Tr(K_{x}^{*}K_{x})\leq\kappa,\hskip 8.5359pt\forall
x\in\mathcal{X}.$ (7)

Moreover we assume that the norms $\left\lVert
f_{k}\right\rVert_{\mathcal{H}}$, $k=1,2,\ldots,l,$ are under our control,
such that we can put a threshold $\gamma_{l}>0$ and consider $\left\lVert
f_{k}\right\rVert_{\mathcal{H}}\leq\gamma_{l}.$

##### Further useful observations

Then we have

$\displaystyle K^{*}_{t}K_{x}=K(t,x)\in\mathcal{L}_{2}(\mathcal{Y})\hskip
8.5359pt\forall x,t\in\mathcal{X}.$ (8)

Given $x\in\mathcal{X}$ the operator

$\displaystyle T_{x}=K_{x}K_{x}^{*}\in\mathcal{L}_{2}(\mathcal{H}),$ (9)

is a positive Hilbert-Schmidt operator and (9) ensures

$\displaystyle\left\lVert
T_{x}\right\rVert_{\mathcal{L}(\mathcal{H})}\leq\left\lVert
T_{x}\right\rVert_{\mathcal{L}_{2}(\mathcal{H})}=\left\lVert
K(x,x)\right\rVert_{\mathcal{L}_{2}(\mathcal{Y})}\leq\kappa.$ (10)

Let $T_{q_{\mathcal{X}}}:\mathcal{H}\rightarrow\mathcal{H}$ be

$T_{q_{\mathcal{X}}}=\int_{\mathcal{X}}T_{x}\mathop{}\\!\mathrm{d}q_{\mathcal{X}}(x),$

where the integral converges in $\mathcal{L}_{2}(\mathcal{H})$ to a positive
trace class operator with

$\displaystyle\left\lVert
T_{q_{\mathcal{X}}}\right\rVert_{\mathcal{L}(\mathcal{H})}\leq\left\lVert
T_{q_{\mathcal{X}}}\right\rVert_{\mathcal{L}_{2}(\mathcal{H})}\leq
Tr(T_{q_{\mathcal{X}}})=\int_{\mathcal{X}}Tr(T_{x})\mathop{}\\!\mathrm{d}q_{\mathcal{X}}(x)\leq\kappa.$
(11)

Following Proposition 1 in Caponnetto & De Vito (2007), we have the minimizers
$f_{q}$ of expected risk $\mathcal{R}_{q}$ are the solution of the following
equation:

$\displaystyle T_{q_{\mathcal{X}}}f_{q}=g,$

where

$g=\int_{\mathcal{X}}K_{x}f_{q}(x)\mathop{}\\!\mathrm{d}q_{\mathcal{X}}(x)\in\mathcal{H},$

with integral converging in $\mathcal{H}.$

Next we define the operators

$\displaystyle
T_{\mathbf{x}^{\prime}}=\frac{1}{m}\sum_{j=1}^{m}K_{x_{j}^{\prime}}K_{x_{j}^{\prime}}^{*},$
$\displaystyle
T_{\mathbf{x},\beta}=\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})K_{x_{i}}K_{x_{i}}^{*},$
$\displaystyle
g_{\mathbf{x},{\mathbf{y}},\beta}=\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})K_{x_{i}}y_{i}.$

In the sequel we adopt the convention that $C$ denotes a generic positive
coefficient, which can vary from appearance to appearance and may only depend
on basic parameter such as $p_{\mathcal{X}}$, $q_{\mathcal{X}}$, $\kappa$,
$B$, $y_{0}$ and others introduced below, but not on $n,m$ and error
probability $\delta>0$.

We will need the following statements.

######  Lemma 1.

With probability at least $1-\delta$ we have

$\displaystyle\left\lVert
T_{q_{\mathcal{X}}}-T_{\mathbf{x}^{\prime}}\right\rVert_{\mathcal{L}(\mathcal{H})}\leq\left\lVert
T_{q_{\mathcal{X}}}-T_{\mathbf{x}^{\prime}}\right\rVert_{\mathcal{L}_{2}(\mathcal{H})}\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)m^{-\frac{1}{2}},$ (12)
$\displaystyle\left\lVert
T_{\mathbf{x}^{\prime}}-T_{\mathbf{x},\beta}\right\rVert_{\mathcal{L}(\mathcal{H})}\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)\left(n^{-\frac{1}{2}}+m^{-\frac{1}{2}}\right),$
(13) $\displaystyle\left\lVert
T_{\mathbf{x},\beta}f^{*}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rVert_{\mathcal{H}}\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)n^{-\frac{1}{2}},$ (14)

where $C>0$ does not depend on $n,m$ and $\delta.$

The proof of Lemma 1 is based on Lemma 4 of Huang et al. (2006), which we
formulate in our notations as follows

######  Lemma 2.

((Huang et al., 2006)) Let $\phi$ be a map from $\mathbf{U}$ to $\mathbf{U}$
such that $\left\lVert\phi(x)\right\rVert_{\mathbf{U}}\leq R$ for all
$x\in\mathcal{X}$. Then with probability at least $1-\delta$ it holds

$\left\lVert\frac{1}{m}\sum_{j=1}^{m}\phi(x_{j}^{\prime})-\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})\phi(x_{i})\right\rVert_{\mathbf{U}}\leq\left(1+\sqrt{2\log\frac{2}{\delta}}\right)R\sqrt{\frac{B^{2}}{n}+\frac{1}{m}}.$

Moreover, we will need a concentration inequality that follows from Pinelis
(1992), see also Rosasco et al. (2010).

######  Lemma 3 (Concentration lemma).

If $\xi_{1},\xi_{2},\ldots,\xi_{n}$ are zero mean independent random variables
with values in a separable Hilbert space $\mathbf{U}$, and for some $D>0$ one
has $\left\lVert\xi_{i}\right\rVert_{\mathbf{U}}\leq D$, $i=1,2,\ldots,n,$
then the following bound

$\left\lVert\frac{1}{n}\sum_{i=1}^{n}\xi_{i}\right\rVert_{\mathbf{U}}\leq\frac{D\sqrt{2\log\frac{2}{\delta}}}{\sqrt{n}}$

holds true with probability at least $1-\delta$.

Proof of Lemma 1.

Let us start by proving (12) by introducing the map
$\xi:\mathcal{X}\rightarrow\mathcal{L}_{2}(\mathcal{H})$ as
$\xi(x)=K_{x}K_{x}^{*}-T_{q_{\mathcal{X}}}.$ From (10) and (11) it follows
that

$\left\lVert\xi(x)\right\rVert_{\mathcal{L}_{2}(\mathcal{H})}\leq\left\lVert
K_{x}K_{x}^{*}\right\rVert_{\mathcal{L}_{2}(\mathcal{H})}+\left\lVert
T_{q_{\mathcal{X}}}\right\rVert_{\mathcal{L}_{2}(\mathcal{H})}\leq 2\kappa.$

Moreover, we have

$\int_{\mathcal{X}}\xi(x)dq_{\mathcal{X}}(x)=\int_{\mathcal{X}}K_{x}K_{x}^{*}dq_{\mathcal{X}}(x)-T_{q_{\mathcal{X}}}=0.$

Therefore, for $x_{j}^{\prime},j=1,2,\ldots,m,$ drawn i.i.d from the marginal
probability measure $q_{\mathcal{X}},$ the corresponding operators
$\xi_{j}=\xi(x_{j}^{\prime})$ can be treated as zero mean independent random
variables in ${\mathcal{L}_{2}(\mathcal{H})},$ such that the condition of
Concentration lemma are satisfied with $D=2\kappa,$ and

$\left\lVert
T_{\mathbf{x}^{\prime}}-T_{q_{\mathcal{X}}}\right\rVert_{\mathcal{L}_{2}(\mathcal{H})}=\left\lVert\frac{1}{m}\sum_{j=1}^{m}K_{x_{j}^{\prime}}K_{x_{j}^{\prime}}^{*}-T_{q_{\mathcal{X}}}\right\rVert_{\mathcal{L}_{2}(\mathcal{H})}=\left\lVert\frac{1}{m}\sum_{j=1}^{m}\xi_{j}\right\rVert_{\mathcal{L}_{2}(\mathcal{H})}\leq\frac{2\kappa\sqrt{2\log\frac{2}{\delta}}}{\sqrt{m}}.$

To obtain (13), for any $f\in\mathcal{H}$ we define a map
$\phi=\phi_{f}:\mathcal{X}\rightarrow\mathcal{H}$ as
$\phi_{f}(x)=K_{x}K_{x}^{*}f$. It clear that

$\left\lVert\phi_{f}(x)\right\rVert_{\mathcal{H}}=\left\lVert
K_{x}K_{x}^{*}\right\rVert_{\mathcal{L}(\mathcal{H})}\left\lVert
f\right\rVert_{\mathcal{H}}\leq\kappa\left\lVert f\right\rVert_{\mathcal{H}}.$

Therefore, for the map $\phi=\phi_{f}$ the condition of the above Lemma 2 is
satisfied with $R=\kappa\left\lVert f\right\rVert_{\mathcal{H}}$. Then
directly from that lemma for any $f\in\mathcal{H}$ we have

$\displaystyle\left\lVert
T_{\mathbf{x}^{\prime}}f-T_{\mathbf{x},\beta}f\right\rVert_{\mathcal{H}}$
$\displaystyle=\left\lVert\frac{1}{m}\sum_{j=1}^{m}\phi_{f}(x_{j}^{\prime})-\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})\phi_{f}(x_{i})\right\rVert_{\mathcal{H}}$
$\displaystyle\leq\left(1+\sqrt{2\log\frac{2}{\delta}}\right)\left(\sqrt{\frac{B^{2}}{n}+\frac{1}{m}}\right)\kappa\left\lVert
f\right\rVert_{\mathcal{H}}$ $\displaystyle\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)\left(m^{-\frac{1}{2}}+n^{-\frac{1}{2}}\right)\left\lVert
f\right\rVert_{\mathcal{H}},$

that proves (13).

Consider now the map
$\digamma:\mathcal{X}\times\mathcal{Y}\rightarrow\mathcal{H}$ defined by

$\digamma(x,y)=\beta(x)K_{x}(f_{p}(x)-y).$

Recall that $\left\lVert
K_{x}\right\rVert_{\mathcal{L}(\mathcal{Y},\mathcal{H})}\leq\sqrt{Tr(K_{x}^{*}K_{x})}\leq\sqrt{\kappa}$.
Then we obtain:

$\left\lVert\digamma(x,y)\right\rVert_{\mathcal{H}}\leq\left\lVert
K_{x}\right\rVert_{\mathcal{L}(\mathcal{Y},\mathcal{H})}\left\lVert\int_{\mathcal{Y}}y^{\prime}dp(y^{\prime}|x)-y\right\rVert_{\mathcal{Y}}\left|\beta(x)\right|\leq
2y_{0}B\sqrt{\kappa}.$

Moreover, for $p(x,y)=p(y|x)p_{\mathcal{X}}(x)$ we have

$\displaystyle\int_{\mathcal{X}\times\mathcal{Y}}\digamma(x,y)dp(x,y)$
$\displaystyle=\int_{\mathcal{X}}K_{x}\beta(x)\int_{\mathcal{Y}}\left(\int_{\mathcal{Y}}y^{\prime}dp(y^{\prime}|x)-y\right)dp(y|x)dp_{\mathcal{X}}(x)=0,$

such that for $(x_{i},y_{i}),$ $i=1,2,\ldots,n$, drawn i.i.d from the measure
$p(x,y)$ the corresponding values $\digamma_{i}=\digamma(x_{i},y_{i})$ are
zero mean independent random variables in $\mathcal{H}$.
Then for the just defined
$\digamma_{i}=\beta(x_{i})K_{x_{i}}(f_{q}(x_{i})-y_{i})$ the conditions of
Lemma 3 are satisfied with $D=2y_{0}B\sqrt{\kappa},$ such that

$\displaystyle\left\lVert\frac{1}{n}\sum_{i=1}^{n}\digamma_{i}\right\rVert_{\mathcal{H}}$
$\displaystyle=\left\lVert\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})K_{x_{i}}(f_{q}(x_{i})-y_{i})\right\rVert_{\mathcal{H}}$
$\displaystyle=\left\lVert\sum_{i=1}^{n}\beta(x_{i})K_{x_{i}}K_{x_{i}}^{*}f_{q}-\sum_{i=1}^{n}\beta(x_{i})K_{x_{i}}y_{i}\right\rVert_{\mathcal{H}}$
$\displaystyle=\left\lVert
T_{\mathbf{x},\beta}f_{q}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rVert_{\mathcal{H}}\leq\frac{2y_{0}B\sqrt{\kappa}\sqrt{2\log\frac{2}{\delta}}}{\sqrt{n}}.$

This bound gives us (14).

##### Aggregation for vector-valued functions

Next we construct a new approximant in the form of a linear combination of
approximants $f_{1},f_{2},\ldots,f_{l},$ computed for all tried parameter
values. The linear combination of the approximants is computed as

$\displaystyle f=\sum_{k=1}^{l}c_{k}f_{k}.$ (15)

Since $f_{1},f_{2},\ldots,f_{l}$ belong to RKHS $\mathcal{H}$, it is clear
that $f\in\mathcal{H}.$ Now we want to argue on how close we can get to
$f_{q}$. Following Proposition 1 in Caponnetto & De Vito (2007), we have

$\displaystyle\mathcal{R}_{q}(f)-\mathcal{R}_{q}(f_{q})=\left\lVert
f-f_{q}\right\rVert_{L^{2}(q_{\mathcal{X}})}^{2}=\left\lVert\sqrt{T_{q_{\mathcal{X}}}}(f-f_{q})\right\rVert_{\mathcal{H}}^{2}.$
(16)

Next we observe that the best approximation $f^{*}$ of the target regression
function $f_{q}$ by linear combinations corresponds to the vector
$c^{*}=(c_{1}^{*},\ldots,c_{l}^{*})$ of ideal coefficients in (15) that solves
the linear system $Gc^{*}=\bar{g}$ with the Gram matrix
$G=\left(\left\langle\sqrt{T_{q_{\mathcal{X}}}}f_{k},\sqrt{T_{q_{\mathcal{X}}}}f_{u}\right\rangle_{\mathcal{H}}\right)_{k,u=1}^{l}$
and the right-hand side vector
$\bar{g}=\left(\left\langle\sqrt{T_{q_{\mathcal{X}}}}f_{q},\sqrt{T_{q_{\mathcal{X}}}}f_{k}\right\rangle_{\mathcal{H}}\right)_{k=1}^{l}$.
Let us provide a prove of this short observation in the next lemma. Note that
the entries $G$ and $g$ can equivalently also be formulated in terms of
$\left\langle.,.\right\rangle_{L^{2}(q_{\mathcal{X}})}$, as done in the main
text. We are going to use this formulation in the next lemma in order to be
compatible with the main text (switching to the inner products in terms of
$\mathcal{H}$ would not change the argument of the proof at all):

######  Lemma 4.

The best $L^{2}(q_{\mathcal{X}})$-approximation $f^{*}$ of the target
regression function $f_{q}$ by linear combinations corresponds to the vector
$c^{*}=(c_{1}^{*},\ldots,c_{l}^{*})=G^{-1}\overline{g}$.

###### Proof.

Let us denote (16) by $f(c)$ and rewrite this expression appropriately:

$\displaystyle f(c)=\sum_{i,j=1}^{l}c_{i}c_{j}\left\langle
f_{i},f_{j}\right\rangle_{L^{2}(q_{\mathcal{X}})}-2\sum_{i=1}^{l}c_{i}\left\langle
f_{i},f_{q}\right\rangle_{L^{2}(q_{\mathcal{X}})}+\left\langle
f_{q},f_{q}\right\rangle_{L^{2}(q_{\mathcal{X}})}.$

Taking the derivative with respect to $c_{i}$ yield:

$\displaystyle\frac{\partial f(c)}{\partial
c_{i}}=2\left(\sum_{j=1}^{l}c_{j}\left\langle
f_{i},f_{j}\right\rangle_{L^{2}(q_{\mathcal{X}})}-\left\langle
f_{i},f_{q}\right\rangle_{L^{2}(q_{\mathcal{X}})}\right).$

Setting these derivatives to zero (for all $i\in\\{1,\ldots,l\\}$) gives the
claimed equation. Noting that the Hessian is equal to $2G$ (and thus positive-
definite) ensures that $c^{*}$ is a global minimum of $f$. ∎

But, of course, neither Gram matrix $G$ nor the vector $\bar{g}$ is
accessible, because there is no access to the target measure
$q_{\mathcal{X}}$, so we switch to the empirical counterparts $\widetilde{G}$
and $\widetilde{g}$.

Then the following lemma is helpful to gain some information on the error made
by the empirical average:

######  Lemma 5.

With probability $1-\delta$ we have

$\displaystyle\bigg{|}\left\langle\sqrt{T_{q_{\mathcal{X}}}}f_{u},\sqrt{T_{q_{\mathcal{X}}}}f_{k}\right\rangle_{\mathcal{H}}-\frac{1}{m}\sum_{j=1}^{m}\left\langle
f_{k}(x^{\prime}_{j}),f_{u}(x^{\prime}_{j})\right\rangle_{\mathcal{Y}}\bigg{|}\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)m^{-\frac{1}{2}},$
$\displaystyle\bigg{|}\left\langle\sqrt{T_{q_{\mathcal{X}}}}f_{k},\sqrt{T_{q_{\mathcal{X}}}}f_{q}\right\rangle_{\mathcal{H}}-\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})\left\langle
f_{k}(x_{i}),y_{i}\right\rangle_{\mathcal{Y}}\bigg{|}\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)(n^{-\frac{1}{2}}+m^{-\frac{1}{2}}),$

where $C>0$ does not depend on $n,m$ and $\delta.$

###### Proof.

Keeping in mind that $f_{q},f_{k}\in{\mathcal{H}}$ we have

$\displaystyle\left\langle\sqrt{T_{q_{\mathcal{X}}}}f_{u},\sqrt{T_{q_{\mathcal{X}}}}f_{k}\right\rangle_{\mathcal{H}}$
$\displaystyle=\left\langle
T_{\mathbf{x}^{\prime}}f_{k},f_{u}\right\rangle_{\mathcal{H}}+\left\langle(T_{q_{\mathcal{X}}}-T_{\mathbf{x}^{\prime}})f_{u},f_{k}\right\rangle_{\mathcal{H}}$
$\displaystyle=\left\langle\frac{1}{m}\sum_{j=1}^{m}K_{x^{\prime}_{j}}K_{x^{\prime}_{j}}^{*}f_{k},f_{u}\right\rangle_{\mathcal{H}}+\left\langle(T_{q_{\mathcal{X}}}-T_{\mathbf{x}^{\prime}})f_{u},f_{k}\right\rangle_{\mathcal{H}}$
$\displaystyle=\frac{1}{m}\sum_{j=1}^{m}\left\langle
K_{x^{\prime}_{j}}^{*}f_{k},K_{x^{\prime}_{j}}^{*}f_{u}\right\rangle_{\mathcal{Y}}+\left\langle(T_{q_{\mathcal{X}}}-T_{\mathbf{x}^{\prime}})f_{u},f_{k}\right\rangle_{\mathcal{H}}$
$\displaystyle=\frac{1}{m}\sum_{j=1}^{m}\left\langle
f_{k}(x^{\prime}_{j}),f_{u}(x^{\prime}_{j})\right\rangle_{\mathcal{Y}}+\left\langle(T_{q_{\mathcal{X}}}-T_{\mathbf{x}^{\prime}})f_{u},f_{k}\right\rangle_{\mathcal{H}}.$

Moreover, from (12) with probability $1-\delta$ we have that

$\displaystyle\bigg{|}\left\langle(T_{q_{\mathcal{X}}}-T_{\mathbf{x}^{\prime}})f_{u},f_{k}\right\rangle_{\mathcal{H}}\bigg{|}\leq
C\left\lVert f_{u}\right\rVert_{\mathcal{H}}\left\lVert
f_{k}\right\rVert_{\mathcal{H}}\left(log^{\frac{1}{2}}\frac{1}{\delta}\right)m^{-\frac{1}{2}}.$

Then

$\displaystyle\bigg{|}\left\langle\sqrt{T_{q_{\mathcal{X}}}}f_{u},\sqrt{T_{q_{\mathcal{X}}}}f_{k}\right\rangle_{\mathcal{H}}-\frac{1}{m}\sum_{j=1}^{m}\left\langle
f_{k}(x^{\prime}_{j}),f_{u}(x^{\prime}_{j})\right\rangle_{\mathcal{Y}}\bigg{|}\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)m^{-\frac{1}{2}}.$

Now, we prove the second statement in Lemma 5. We have

$\displaystyle\left\langle\sqrt{T_{q_{\mathcal{X}}}}f_{k},\sqrt{T_{q_{\mathcal{X}}}}f_{q}\right\rangle_{\mathcal{H}}$
$\displaystyle=\left\langle
f_{k},T_{q_{\mathcal{X}}}f_{q}\right\rangle_{\mathcal{H}}=\left\langle
f_{k},T_{q_{\mathcal{X}}}f_{q}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rangle_{\mathcal{H}}+\left\langle
f_{k},g_{\mathbf{x},{\mathbf{y}},\beta}\right\rangle_{\mathcal{H}}$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})\left\langle
f_{k},K_{x_{i}}y_{i}\right\rangle_{\mathcal{H}}+\left\langle
f_{k},T_{q_{\mathcal{X}}}f_{q}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rangle_{\mathcal{H}}$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})\left\langle
K_{x_{i}}^{*}f_{k},y_{i}\right\rangle_{\mathcal{Y}}+\left\langle
f_{k},T_{q_{\mathcal{X}}}f_{q}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rangle_{\mathcal{H}}$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})\left\langle
f_{k}(x_{i}),y_{i}\right\rangle_{\mathcal{Y}}+\left\langle
f_{k},T_{q_{\mathcal{X}}}f_{q}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rangle_{\mathcal{H}}.$

From Lemma 1, with probability $1-\delta$ we have

$\displaystyle\left\lVert
T_{q_{\mathcal{X}}}f_{q}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rVert_{\mathcal{H}}$
$\displaystyle\leq\left\lVert
T_{q_{\mathcal{X}}}f_{q}-T_{\mathbf{x}^{\prime}}f_{q}\right\rVert_{\mathcal{H}}+\left\lVert
T_{\mathbf{x}^{\prime}}f_{q}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rVert_{\mathcal{H}}$
$\displaystyle\leq\left\lVert
T_{q_{\mathcal{X}}}f_{q}-T_{\mathbf{x}^{\prime}}f_{q}\right\rVert_{\mathcal{H}}+\left\lVert
T_{\mathbf{x}^{\prime}}f_{q}-T_{\mathbf{x},\beta}f_{q}\right\rVert_{\mathcal{H}}+\left\lVert
T_{\mathbf{x},\beta}f_{q}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rVert_{\mathcal{H}}$
$\displaystyle\leq C\left\lVert
f_{q}\right\rVert_{\mathcal{H}}\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)m^{-\frac{1}{2}}+C\left\lVert
f_{q}\right\rVert_{\mathcal{H}}\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)(n^{-\frac{1}{2}}+m^{-\frac{1}{2}})+\left\lVert
T_{\mathbf{x},\beta}f_{q}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rVert_{\mathcal{H}}$
$\displaystyle\leq C\left\lVert
f_{q}\right\rVert_{\mathcal{H}}\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)m^{-\frac{1}{2}}+C\left\lVert
f_{q}\right\rVert_{\mathcal{H}}\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)(n^{-\frac{1}{2}}+m^{-\frac{1}{2}})+C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)n^{-\frac{1}{2}}.$

Then

$\displaystyle\left\langle
f_{k},T_{q_{\mathcal{X}}}f_{q}-g_{\mathbf{x},{\mathbf{y}},\beta}\right\rangle_{\mathcal{H}}\leq
C\left\lVert
f_{k}\right\rVert_{\mathcal{H}}\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)\left(n^{-\frac{1}{2}}+m^{-\frac{1}{2}}\right).$

Therefore,

$\displaystyle\bigg{|}\left\langle\sqrt{T_{q_{\mathcal{X}}}}f_{k},\sqrt{T_{q_{\mathcal{X}}}}f_{q}\right\rangle_{\mathcal{H}}-\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})\left\langle
f_{k}(x_{i}),y_{i}\right\rangle_{\mathcal{Y}}\bigg{|}\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)(n^{-\frac{1}{2}}+m^{-\frac{1}{2}}).$

∎

##### Towards our main generalization bound

Next we use similar arguments as in Theorem 4 from Gizewski et al. (2022) to
obtain our main result, Theorem 1. Lemma 5 suggests to approximate $G$ and
$\bar{g}$ by their empirical counterparts:

$\displaystyle\tilde{G}$
$\displaystyle=\left(\frac{1}{m}\sum_{j=1}^{m}\left\langle
f_{k}(x_{j}^{{}^{\prime}}),f_{u}(x_{j}^{\prime})\right\rangle_{\mathcal{Y}}\right)_{k,u=1}^{l},$
(17) $\displaystyle\tilde{g}$
$\displaystyle=\left(\frac{1}{n}\sum_{i=1}^{n}\beta(x_{i})\left\langle
y_{i},f_{k}(x_{i})\right\rangle_{\mathcal{Y}}\right)_{k=1}^{l}$ (18)

which can be effectively computed from data samples. Moreover, again from
Lemma 5 we can argue that with probability $1-\delta$ it holds:

$\displaystyle\|\bar{g}-\tilde{g}\|_{\mathbb{R}^{l}}\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)(n^{-\frac{1}{2}}+m^{-\frac{1}{2}}),$
(19) $\displaystyle\|G-\tilde{G}\|_{\mathcal{L}(\mathbb{R}^{l})}\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)m^{-\frac{1}{2}}.$ (20)

With the matrix $\tilde{G}$ at hand one can easily check whether or not it is
well-conditioned and $\tilde{G}^{-1}$ exists (otherwise one needs to get rid
of models with similar performance). Thus the norms
$\left\lVert\tilde{G}\right\rVert_{\mathcal{L}(\mathbb{R}^{l})}$ and
$\left\lVert\tilde{G}^{-1}\right\rVert_{\mathcal{L}(\mathbb{R}^{l})}$ can be
bounded independently of $m$ and $n$, due to the fact that all their entries
can be bounded as follows (we only do the calculation for the entries of
$\tilde{G}$):

$\displaystyle|\tilde{G}_{k,u}|$
$\displaystyle\leq\frac{1}{m}\sum_{j=1}^{m}\left|\left\langle
f_{k}(x_{j}^{{}^{\prime}}),f_{u}(x_{j}^{\prime})\right\rangle_{\mathcal{Y}}\right|=\frac{1}{m}\sum_{j=1}^{m}\left|\left\langle
K^{*}_{x_{j}^{{}^{\prime}}}f_{k},K^{*}_{x_{j}^{{}^{\prime}}}f_{u}\right\rangle_{\mathcal{Y}}\right|$
$\displaystyle=\frac{1}{m}\sum_{j=1}^{m}\left|\left\langle
K_{x_{j}^{{}^{\prime}}}K^{*}_{x_{j}^{{}^{\prime}}}f_{k},f_{u}\right\rangle_{\mathcal{H}}\right|=\frac{1}{m}\sum_{j=1}^{m}\left|\left\langle
T_{x_{j}^{{}^{\prime}}}f_{k},f_{u}\right\rangle_{\mathcal{H}}\right|$
$\displaystyle\leq\frac{1}{m}\sum_{j=1}^{m}\left\lVert
T_{x_{j}^{{}^{\prime}}}\right\rVert_{{}_{\mathcal{L}(\mathcal{H})}}\left\lVert
f_{k}\right\rVert_{\mathcal{H}}\left\lVert
f_{u}\right\rVert_{\mathcal{H}}\leq\kappa\gamma_{l}^{2},$

where we used the reproducing property (5) to obtain the equality in the first
line and (10) for the last inequality. Now assume that $m$ is so large that
with probability $1-\delta$ we have

$\|G-\tilde{G}\|_{\mathcal{L}(\mathbb{R}^{l})}<\frac{1}{\left\|\tilde{G}^{-1}\right\|_{\mathcal{L}(\mathbb{R}^{l})}}.$
(21)

Moreover we can use the following simple manipulation:

$\displaystyle
G^{-1}=\tilde{G}^{-1}(G\tilde{G}^{-1})^{-1}=\tilde{G}^{-1}(I-(I-G\tilde{G}^{-1}))^{-1}=\tilde{G}^{-1}(I-(\tilde{G}-G)\tilde{G}^{-1})^{-1}.$

Then (21) ensures that the Neumann series for
$(I-(\tilde{G}-G)\tilde{G}^{-1})^{-1}$ converges and we obtain the following
bound:

$\displaystyle\left\|G^{-1}\right\|_{\mathcal{L}(\mathbb{R}^{l})}\leq\frac{\left\|\tilde{G}^{-1}\right\|_{\mathcal{L}(\mathbb{R}^{l})}}{1-\left\|\tilde{G}^{-1}\right\|_{\mathcal{L}(\mathbb{R}^{l})}\left\|G-\tilde{G}\right\|_{\mathcal{L}(\mathbb{R}^{l})}}=O(1).$
(22)

Now we are in the position to prove our main generalization bound (4) for
unsupervised domain adaptation:

###### Proof of Theorem 1.

We have already discussed that the coefficients in the best approximation
$f^{*}$ to $f_{q}$ are given by $c^{*}=$
$\left(c^{*}_{1},c^{*}_{2},\ldots,c^{*}_{l}\right)=G^{-1}\bar{g}.$ Since:

$\displaystyle
G^{-1}(\tilde{g}-\bar{g})+G^{-1}(G-\tilde{G})\tilde{c}=G^{-1}\tilde{g}-c^{*}+\tilde{c}-G^{-1}\tilde{g}=\tilde{c}-c^{*}$

then from (19)–(22) with probability $1-\delta$ we have

$\displaystyle\|\tilde{c}-c^{*}\|_{\mathbb{R}^{l}}=$
$\displaystyle\leq\left\|G^{-1}\right\|_{\mathcal{L}(\mathbb{R}^{l})}\left(\|\tilde{g}-\bar{g}\|_{\mathbb{R}^{l}}+\|G-\tilde{G}\|_{\mathcal{L}(\mathbb{R}^{l})}\|\tilde{c}\|_{\mathbb{R}^{l}}\right)$
$\displaystyle\leq
C\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)(n^{-\frac{1}{2}}+m^{-\frac{1}{2}}).$
(23)

Moreover:

$\displaystyle\mathcal{R}_{q}(\tilde{f})-\mathcal{R}_{q}(f_{q})$
$\displaystyle=\left\lVert\sqrt{T_{q_{\mathcal{X}}}}(\tilde{f}-f_{q})\right\rVert_{\mathcal{H}}^{2}$
$\displaystyle\leq\left(\left\lVert\sqrt{T_{q_{\mathcal{X}}}}(f^{*}-f_{q})\right\rVert_{\mathcal{H}}+\left\lVert\sqrt{T_{q_{\mathcal{X}}}}(\tilde{f}-f^{*})\right\rVert_{\mathcal{H}}\right)^{2}$
$\displaystyle\leq
2\left\lVert\sqrt{T_{q_{\mathcal{X}}}}(f^{*}-f_{q})\right\rVert_{\mathcal{H}}^{2}+2\left\lVert\sqrt{T_{q_{\mathcal{X}}}}(\tilde{f}-f^{*})\right\rVert_{\mathcal{H}}^{2}$
$\displaystyle=2\left(\mathcal{R}_{q}\left(f^{*}\right)-\mathcal{R}_{q}(f_{q})\right)+2\left\lVert\sqrt{T_{q_{\mathcal{X}}}}(\tilde{f}-f^{*})\right\rVert_{\mathcal{H}}^{2}$
$\displaystyle\leq
2\left(\mathcal{R}_{q}\left(f^{*}\right)-\mathcal{R}_{q}(f_{q})\right)+2\left(\sum_{k=1}^{l}|c^{*}_{k}-\tilde{c}_{k}|\left\lVert\sqrt{T_{q_{\mathcal{X}}}}f_{k}\right\rVert_{\mathcal{H}}\right)^{2}$
$\displaystyle\leq
2\left(\mathcal{R}_{q}\left(f^{*}\right)-\mathcal{R}_{q}(f_{q})\right)+2l\|c^{*}-\tilde{c}\|_{\mathbb{R}^{l}}^{2}\max_{k}\left\|\sqrt{T_{q_{\mathcal{X}}}}f_{k}\right\|_{\mathcal{H}}^{2}$
$\displaystyle\leq
2\left(\mathcal{R}_{q}\left(f^{*}\right)-\mathcal{R}_{q}(f_{q})\right)+2\left\lVert\sqrt{T_{q_{\mathcal{X}}}}\right\rVert_{\mathcal{L}(\mathcal{H})}^{2}l\gamma_{l}^{2}\|c^{*}-\tilde{c}\|_{\mathbb{R}^{l}}^{2},$
(24)

The statement of the theorem follows now from (23)–(24) (using again the
inequality $(a+b)^{2}\leq 2(a^{2}+b^{2})$). ∎

##### On the dependence of the error bound on the number $l$ of models

An interesting question is, how the bound in Eq. (4) depends on $l$. To this
end, let us have a look at the second term in the last line in Eq. (24):
$\left\lVert\sqrt{T_{q_{\mathcal{X}}}}\right\rVert_{\mathcal{L}}$ analyzes a
sampling operator, thus does not depend on the number of models, same goes for
$\gamma_{l}$, which is just a uniform bound on all our models. To analyze
$\|c^{*}-\tilde{c}\|_{\mathbb{R}^{l}}^{2}$, let us have a look at the
individual factors in the first inequality of Eq. (23). To not overload
notation, $C>0$ is used here for any absolute constant that is independent of
$l,m,n$ and $\delta$.

  * •

$\|G-\tilde{G}\|_{\mathcal{L}(\mathbb{R}^{l})}$: The individual entries of
$G-\tilde{G}$ are (in absolute values) bounded by Lemma 5. The proof arguments
only involve norm bounds on the associated sampling operators, the uniform
bound $\gamma_{l}$ on all the models and the bound $B$ on $\beta$, thus the
absolute constant $C$ there is independent of $l$. By the definition of the
matrix Frobenius norm, we thus get

$\|G-\tilde{G}\|_{\mathcal{L}(\mathbb{R}^{l})}\leq
Cl\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)(n^{-\frac{1}{2}}+m^{-\frac{1}{2}})$
(25)

  * •

$\|\tilde{g}-\bar{g}\|_{\mathbb{R}^{l}}$: Similar arguments as before lead to
$\|\tilde{g}-\bar{g}\|_{\mathcal{L}(\mathbb{R}^{l})}\leq
C\sqrt{l}\left(\log^{\frac{1}{2}}\frac{1}{\delta}\right)(n^{-\frac{1}{2}}+m^{-\frac{1}{2}})$

  * •

$\left\|G^{-1}\right\|_{\mathcal{L}(\mathbb{R}^{l})}$: It is natural to assume
that there is some constant
$c\geq\left\|\tilde{G}^{-1}\right\|_{\mathcal{L}(\mathbb{R}^{l})}$. Otherwise,
we can, e.g., orthogonalize our models and coefficients without changing the
aggregation, but with reducing the conditioning number (i.e., with reducing
$\left\|\tilde{G}^{-1}\right\|$). It is also natural to assume that $m$ and
$n$ are large enough such that
$l(n^{-\frac{1}{2}}+m^{-\frac{1}{2}})<\frac{1}{2c}$. Then applying Eq. (25) to
Eq. (22), we can deduce that
$\left\|G^{-1}\right\|_{\mathcal{L}(\mathbb{R}^{l})}\leq 2c$.

  * •

$\|\tilde{c}\|_{\mathbb{R}^{l}}$: This quantity can also be assumed to be
known independently of $l$, since it is given by our data only.

Combining the previous points gives us
$\|c^{*}-\tilde{c}\|_{\mathbb{R}^{l}}^{2}\leq
Cl^{2}\left(\log\frac{1}{\delta}\right)(n^{-1}+m^{{-1}})$ which finally leads
to the refined bound

$\displaystyle\mathcal{R}_{q}(\tilde{f})-\mathcal{R}_{q}(f_{q})\leq
2\left(\mathcal{R}_{q}\left(f^{*}\right)-\mathcal{R}_{q}(f_{q})\right)+Cl^{3}\left(\log\frac{1}{\delta}\right)(n^{-1}+m^{{-1}})$

for sufficiently large $l$, $m$ and $n$ and error probability $\delta>0$.

##  Appendix B Construction of Function Spaces

Let us give a short discussion on the construction of our required function
space mentioned in the previous Section A, the reproducing kernel space
$\mathcal{H}$. As mentioned already in the main text, the explicit knowledge
of $\mathcal{H}$ is not required, we just need to rely on its existence.
First, any of our models $f$ can be regarded as an element of some reproducing
kernel space (RKHS) $\tilde{\mathcal{H}}$ satisfying the assumptions 1. This
is immediate if $f:\mathcal{X}\to\mathbb{R}$ is a real valued continuous
function and we take $k(x,y)=f(x)f(y)$ as the associated reproducing kernel.
In the case $f:\mathcal{X}\to\mathcal{Y}$ and $\mathcal{Y}$ is finite
dimensional, it is not hard to see that a similar construction is possible, as
this case can again be boiled down to the construction of a kernel with real-
valued output, see e.g. Remark 1 in Caponnetto & De Vito (2007) for details.

Overall we end up with a finite sequence of spaces
$(\mathcal{H}_{k})_{k=1}^{l+1}$ of functions living on the same domain
$\mathcal{X}$ (we have $l+1$ as we also take into account the regression
function), and the existence of a RKHS containing all given models and the
regression function is not a real restriction. For example, in case of real
valued functions, this assumption is automatically satisfied, as linear
combinations of functions with the same domain which stem from a finite
sequence of RKHSs belong to an RKHS. This follows from a classical result by
N. Aronszajn and R. Godement, see e.g. Pereverzyev (2022, Theorem 1.4.).

There is also ongoing research on constructing function spaces (and especially
associated reproducing kernels) for families of neural networks that are used
in applications, see e.g. Ma & Wu (2022) for ReLU networks, Fermanian et al.
(2021) for recurrent networks and Bietti & Mairal (2017; 2019) for
convolutional neural networks. Incorporating these into our work may lead to
refined generalization bounds that also reflect the nature of our models. We
leave the details open for future work.

##  Appendix C Datasets

This section provides an overview over all applied datasets from language,
image, and time-series domains.

##### Illustrative example:

For the illustrative example (Figure 1 in the main paper) we rely on the
following setting, taken from Shimodaira (2000); Sugiyama et al. (2007); You
et al. (2019): The data points are labelled with $y=\frac{\sin(\pi x)}{\pi x}$
with random noise sampled from the normal distribution
$\mathcal{N}\left(0,\left(\frac{1}{4}\right)^{2}\right)$. Moreover
$p_{\mathcal{X}}\sim\mathcal{N}\left(1,\frac{1}{4}\right),q_{\mathcal{X}}\sim\mathcal{N}\left(2,\left(\frac{1}{4}\right)^{2}\right)$.
The density ratio $\beta$ can be computed analytically and is bounded. We
aggregate several linear models with our approach and compare it to the
optimal linear model, whose coefficients have been evaluated using a computer
algebra system.

##### Academic Dataset

We rely on the Transformed Moons dataset (Zellinger et al., 2021), allowing us
to visualize and address low-dimensional input data. The dataset consists of
two-dimensional input data points forming two classes with a “moon-shaped"
support. The shift from source to target domain is simulated by a
transformation in input space as depicted in Figure 3. The results are shown
in the following table:

Table 2: Mean and standard deviation (after $\pm$) of target classification
accuracy on Transformed Moons dataset over three different random
initialization of model weights and 11 domain adaptation methods.

| Transformed Moons
---|---
|  | Heuristic | Theoretical error guarantees |
Method | SO | TMV | TMR | TCR | SOR | IWV | DEV | IWA (ours) | TB
HoMM | $0.994(\pm 0.003)$ | $0.994(\pm 0.000)$ | ${\it 1.000(\pm 0.003)}$ | $1.000(\pm 0.003)$ | $0.985(\pm 0.012)$ | $0.994(\pm 0.003)$ | ${\bf 0.994(\pm 0.003)}$ | $0.992(\pm 0.000)$ | $0.998(\pm 0.000)$
AdvSKM | $0.985(\pm 0.000)$ | ${\it 1.000(\pm 0.000)}$ | $0.998(\pm 0.000)$ | $0.998(\pm 0.000)$ | $0.987(\pm 0.014)$ | $0.985(\pm 0.000)$ | $0.985(\pm 0.000)$ | ${\bf 1.000(\pm 0.000)}$ | $1.000(\pm 0.000)$
DIRT | $0.964(\pm 0.009)$ | $0.827(\pm 0.018)$ | $0.830(\pm 0.017)$ | $0.981(\pm 0.003)$ | ${\it 0.990(\pm 0.006)}$ | $0.964(\pm 0.009)$ | $0.869(\pm 0.009)$ | ${\bf 0.983(\pm 0.003)}$ | $0.966(\pm 0.000)$
DDC | $0.994(\pm 0.023)$ | ${\it 0.998(\pm 0.000)}$ | $0.996(\pm 0.000)$ | $0.996(\pm 0.000)$ | $0.981(\pm 0.009)$ | $0.994(\pm 0.023)$ | $0.994(\pm 0.023)$ | ${\bf 0.998(\pm 0.000)}$ | $0.998(\pm 0.000)$
CMD | $0.990(\pm 0.006)$ | $0.956(\pm 0.000)$ | $0.964(\pm 0.000)$ | ${\it 0.996(\pm 0.000)}$ | $0.994(\pm 0.006)$ | $0.990(\pm 0.006)$ | $0.990(\pm 0.006)$ | ${\bf 0.996(\pm 0.000)}$ | $1.000(\pm 0.000)$
MMDA | $0.990(\pm 0.006)$ | ${\it 1.000(\pm 0.003)}$ | $1.000(\pm 0.003)$ | $1.000(\pm 0.003)$ | $0.996(\pm 0.009)$ | $0.990(\pm 0.006)$ | $0.990(\pm 0.006)$ | ${\bf 1.000(\pm 0.003)}$ | $1.000(\pm 0.003)$
CoDATS | $0.979(\pm 0.016)$ | ${\it 1.000(\pm 0.044)}$ | $1.000(\pm 0.052)$ | $1.000(\pm 0.016)$ | $0.981(\pm 0.012)$ | $0.979(\pm 0.016)$ | $0.979(\pm 0.174)$ | ${\bf 1.000(\pm 0.010)}$ | $1.000(\pm 0.045)$
Deep-Coral | $1.000(\pm 0.009)$ | ${\it 1.000(\pm 0.000)}$ | $1.000(\pm 0.000)$ | $1.000(\pm 0.000)$ | $0.985(\pm 0.000)$ | $1.000(\pm 0.009)$ | $1.000(\pm 0.009)$ | ${\bf 1.000(\pm 0.000)}$ | $1.000(\pm 0.000)$
CDAN | $1.000(\pm 0.000)$ | ${\it 1.000(\pm 0.000)}$ | $1.000(\pm 0.000)$ | $1.000(\pm 0.000)$ | $0.985(\pm 0.026)$ | $1.000(\pm 0.000)$ | $1.000(\pm 0.000)$ | ${\bf 1.000(\pm 0.000)}$ | $1.000(\pm 0.000)$
DANN | $0.994(\pm 0.006)$ | ${\it 1.000(\pm 0.000)}$ | $1.000(\pm 0.000)$ | $1.000(\pm 0.000)$ | $0.994(\pm 0.016)$ | $0.994(\pm 0.006)$ | $0.994(\pm 0.006)$ | ${\bf 1.000(\pm 0.003)}$ | $1.000(\pm 0.003)$
DSAN | $0.990(\pm 0.009)$ | ${\it 1.000(\pm 0.000)}$ | $1.000(\pm 0.000)$ | $1.000(\pm 0.000)$ | $1.000(\pm 0.003)$ | $0.990(\pm 0.009)$ | $0.990(\pm 0.009)$ | ${\bf 1.000(\pm 0.000)}$ | $1.000(\pm 0.000)$
Avg. | $0.989(\pm 0.008)$ | $0.980(\pm 0.006)$ | $0.981(\pm 0.007)$ | ${\it 0.997(\pm 0.002)}$ | $0.989(\pm 0.010)$ | $0.989(\pm 0.008)$ | $0.981(\pm 0.022)$ | ${\bf 0.997(\pm 0.002)}$ | $0.997(\pm 0.005)$

Figure 3: Transformed Moons dataset. Source data is depicted as blue $+$ and
orange $\times$. Target data points are shown as black dots.

##### Language Dataset

To evaluate our method on a language task, we rely on the Amazon Reviews
(Blitzer et al., 2006) dataset. This dataset consists of text reviews from
four domains: books (B), DVDs (D), electronics (E), and kitchen appliances
(K). Reviews are encoded in $5000$ dimensional feature vectors of bag-of-words
unigrams and bigrams with binary labels: label $0$ if the product is ranked by
$1$ to $3$ stars, and label $1$ if the product is ranked by $4$ or $5$ stars.
From the four categories, we obtain twelve domain adaptation tasks, where each
category serves once as source domain and once as target domain (e.g., see
Table 15). We follow similar data splits as previous works (Chen et al., 2012;
Louizos et al., 2016; Ganin et al., 2016). In particular, we use $4000$
labeled source examples and $4000$ unlabeled target examples for training, and
over $1000$ examples for testing.

##### Image Dataset

Our third dataset is MiniDomainNet, which is based on the DomainNet-2019
dataset (Peng et al., 2019) consisting of six different image domains
(Quickdraw: Q, Real: R, Clipart: C, Sketch: S, Infograph: I, and Painting: P).
We follow Zellinger et al. (2021) and rely on the reduced version of
DomainNet-2019, referred to as MiniDomainNet, which reduces the number of
classes to the top-five largest representatives in the training set across all
six domains. To further improve computation time, we rely on a ImageNet
(Krizhevsky et al., 2012) pre-trained ResNet-18 (He et al., 2016) backbone.
Therefore, we assume that the backbone has learned lower-level filters
suitable for the "Real" image category, and we only need to adapt to the
remaining five domains (e.g., Clipart, Sketch). This results in five domain
adaptation tasks.

##### Time-Series Dataset

We based our time-series experiments on the four datasets included in the
AdaTime benchmark suite (Ragab et al., 2022), which consists of UCI-HAR,
WISDM, HHAR, and Sleep-EDF. The suite includes four representative datasets
spanning $20$ cross-domain real-world scenarios, i.e., human activity
recognition and sleep stage classification. The first dataset is the _Human
Activity Recognition_ (HAR) (Anguita et al., 2013) dataset from the UC Irvine
Repository denoted as UCI-HAR, which contains data from three motion sensors
(accelerometer, gyroscope and body-worn sensors) gathered using smartphones
from $30$ different subjects. It classifies their activities in several
categories, namely, walking, walking upstairs, downstairs, standing, sitting,
and lying down. The WISDM (Kwapisz et al., 2011) dataset is a class-imbalanced
variant from collected accelerometer sensors, including GPS data, from $29$
different subjects which are performing similar activities as in the UCI-HAR
dataset. The _Heterogeneity Human Activity Recognition_ (HHAR) (Stisen et al.,
2015) dataset investigate sensor-, device- and workload-specific
heterogeneities using $36$ smartphones and smartwatches, consisting of $13$
different device models from four manufacturers. Finally, the _Sleep Stage
Classification_ time-series setting aims to classify the
electroencephalography (EEG) signals into five stages i.e., Wake (W), Non-
Rapid Eye Movement stages (N1, N2, N3), and Rapid Eye Movement (REM).
Analogous to Ragab et al. (2022); Eldele et al. (2021), we adopt the Sleep-
EDF-20 dataset obtained from PhysioBank (Goldberger et al., 2000), which
contains EEG readings from $20$ healthy subjects. For all datasets, each
subject is treated as an own domain, and adopt from a source subject to a
target subject.

##  Appendix D Experimental Setup

This section is meant to provide further details on the overall computational
setting of our experiments. We start by giving an overview on the used
computational resources for the specific datasets and the implementation
tools. Next, we describe the network architectures for the individual datasets
in greater detail. In the third subsection we elaborate on the construction of
our models, and the fourth subsection is devoted to matrix inversion. Finally,
in the last subsection, we describe the detailed empirical results and give
the complete tables.

###  D.1 Computational Resources and Implementations

Overall, to compute the results in our tables, we trained $16680$ models with
an approximate computational budget of $1500$ GPU/hours on one high-
performance computing station using $8\times$NVIDIA P$100$ $16$GB, $512$GB
RAM, 40 Cores Xeon(R) CPU E5-2698 v4 @ 2.20GHz on CentOS Linux 7.

Transformed Moons: $11$ methods $\times\ 14$ parameters $\times\ 1$ domain
adaptation tasks $\times\ 3$ seeds $+\ 3$ density estimator classifier $=465$
trained models

Amazon Reviews: $11$ methods $\times\ 14$ parameters $\times\ 12$ domain
adaptation tasks $\times\ 3$ seeds $+\ 12\times 3$ density estimator
classifier $=5580$ trained models

MiniDomainNet: $11$ methods $\times\ 8$ parameters $\times\ 5$ domain
adaptation tasks $\times\ 3$ seeds $+\ 5\times 3$ density estimator classifier
$=1335$ trained models

UCI-HAR: $11$ methods $\times\ 14$ parameters $\times\ 5$ domain adaptation
tasks $\times\ 3$ seeds $+\ 5\times 3$ density estimator classifier $=2325$
trained models

Sleep-EDF: $11$ methods $\times\ 14$ parameters $\times\ 5$ domain adaptation
tasks $\times\ 3$ seeds $+\ 5\times 3$ density estimator classifier $=2325$

HHAR: $11$ methods $\times\ 14$ parameters $\times\ 5$ domain adaptation tasks
$\times\ 3$ seeds $+\ 5\times 3$ density estimator classifier $=2325$ trained
models

WISDM: $11$ methods $\times\ 14$ parameters $\times\ 5$ domain adaptation
tasks $\times\ 3$ seeds $+\ 5\times 3$ density estimator classifier $=2325$
trained models

In Total: $465+5580+1335+4\times 2325=16680$ trained models

All methods have been implemented in Python using the _Pytorch_ (Paszke et
al., 2017, BSD license) library. For monitoring the runs we used Weights &
Biases (Biewald, 2020, MIT license). We use _Scikit-learn_ (Pedregosa et al.,
2011) library for evaluation measures and toy datasets, and the _TQDM_ (da
Costa-Luis, 2019) library, and _Tensorboard_ (Abadi et al., 2015) for keeping
track of the progress of our experiments. We built parts of our implementation
on the codebase of Zellinger et al. (2021, MIT License) and Ragab et al.
(2022, MIT License).

###  D.2 Architectures and Training Setup

In this subsection, we provide details on the model architectures and the
training setup for every dataset. Our base architectures are based on the
AdaTime benchmark suite, which is a large-scale evaluation of domain
adaptation algorithms on time-series data. We extended the benchmark suite to
support $11$ state-of-the-art model architectures on multiple dataset types
ranging from language, image to time-series data, addressed by Transformed
Moons, Amazon Reviews, MiniDomainNet and the four time-series datasets (UCI-
HAR, WISDM, HHAR, and Sleep-EDF) spanning in total $38$ cross-domain real-
world scenarios.

##### Transformed Moons

For the Transformed Moons dataset we use two sequential blocks with fully-
connected layers, 1D-BatchNorm, ReLU activation functions and Dropout. The
full architecture specification can be found in Table 3. The domain classifier
(density ratio estimator) uses the same architecture. We train the class
prediction models for $50$ epochs and the domain classifier for $80$ epochs
with learning rate $0.001$, weight decay $0.0001$ and batchsize $128$ using
the Adam optimizer (Kingma & Ba, 2014). We share the same base architecture
and training setup across every domain adaption method (e.g., DANN, HoMM,
CMD). Additional hyper-parameters are reported in Table 9.

Table 3: Model architecture for the Transformed Moons dataset. The values for
neural network layers correspond to the number of output units.

Architecture
---
| Layers | Values
| Input units | $2$
MLP Block 1 | Fully-connected Layer | $128$
| Batch Normalization 1D Layer | $128$
| ReLU Activation |
| Dropout |
MLP Block 2 | Fully-connected Layer | $128$
| Batch Normalization 1D Layer | $128$
| ReLU Activation |
| Dropout |
| Fully-connected Layer | $128$
| Methods |
| See Table 7 and Table 8 for details. |

##### Amazon Reviews

For the Amazon Reviews dataset we use two sequential blocks with fully-
connected layers, 1D-BatchNorm, ReLU activation function and Dropout,
analogous to the setup for Transformed Moons. We also use the same
architecture for the domain classifier. We train the class prediction models
for $50$ epochs and the domain classifier for $80$ epochs with learning rate
$0.001$, weight decay $0.0001$ and batchsize $128$ using the Adam optimizer
(Kingma & Ba, 2014). We share the same base architecture and training setup
across every domain adaption method (e.g., DANN, HoMM, CMD). Additional hyper-
parameters are reported in Table 9.

Table 4: Model architecture for the Amazon Reviews dataset. The values for
neural network layers correspond to the number of output units.

Architecture
---
| Layers | Values
| Input units | $5000$
MLP Block 1 | Fully-connected Layer | $128$
| Batch Normalization 1D Layer | $128$
| ReLU Activation |
| Dropout |
MLP Block 2 | Fully-connected Layer | $128$
| Batch Normalization 1D Layer | $128$
| ReLU Activation |
| Dropout |
| Fully-connected Layer | $128$
| Methods |
| See Table 7 and Table 8 for details. |

##### MiniDomainNet

Following the pre-trained setup from Peng et al. (2019), we use a frozen
ResNet-18 backbone model which was trained on ImageNet, and operate subsequent
computations on the $512$ dimensional extracted features. To alleviate
overfitting effects on pre-computed features, we perform data augmentation on
the images of each batch and forward each batch through the backbone. We
incorporate zero padding before resizing the images to $256$x$256$ to avoid
image distortions. Furthermore, in alignment with data augmentation techniques
from Shorten & Khoshgoftaar (2019), we perform random resized cropping to
$224$x$224$ with a random viewport between $70\%$ and $100\%$ of the original
image, random horizontal flipping, color jittering of $0.25\%$ on each RGB
channel, and a $\pm 2$ degree rotation.

After the ResNet-18 backbone output, we add a projection layer, and define the
domain adaptation layers on which we use the domain adaptation methods to
align the representations. The backbone and projection layers are defined as a
common architecture across the different domain adaptation methods. Additional
layers are further added for the classification networks, according to the
requirements of the individual domain adaptation methods (e.g., CMD, HoMM).
The number of layers/neurons in the upper layers of our architecture have been
tuned in order to achieve the best performance in the source-only setup. See
Table 5 for a detailed description of the architecture used. We perform
experiments on all $5$ domain adaptation tasks as defined in Section C for
each of the previously listed methods, and with $3$ repetitions based on
different random weights initialization. All class prediction models have been
trained for $60$ epochs and domain classifiers for $100$ epochs with Adam
optimizer, a learning rate of $0.001$, $\beta_{1}=0.9$, $\beta_{2}=0.999$,
batchsize of $128$ and weight decay of $0.0001$. Additional hyper-parameters
are reported in Table 9.

Table 5: Model architecture for the MiniDomainNet dataset. The values for
neural network layers correspond to the number of output units.

Architecture
---
| Layers | Values
Backbone Output Layer | ResNet-18 (Adaptive Average Pooling Layer) | $512$
| Fully-connected Layer | $128$
| Methods |
| See Table 7 and Table 8 for details. |

##### AdaTime

Unless stated otherwise, we follow the implementation and hyper-parameter
settings as reported in Ragab et al. (2022). We extended the AdaTime suite to
comprise a collection of $11$ domain adaptation algorithms. We learned all
domain adaptations models according to the following approaches (see also
Table 6, Table 7 and Table 8): Deep Domain Confusion (DDC) (Tzeng et al.,
2014), Correlation Alignment via Deep Neural Networks (Deep-Coral) (Sun et
al., 2017), Higher-order Moment Matching (HoMM) (Chen et al., 2020), Minimum
Discrepancy Estimation for Deep Domain Adaptation (MMDA) (Rahman et al.,
2020), Central Moment Discrepancy (CMD) (Zellinger et al., 2017), Deep
Subdomain Adaptation (DSAN) (Zhu et al., 2021), Domain-Adversarial Neural
Networks (DANN) (Ganin et al., 2016), Conditional Adversarial Domain
Adaptation (CDAN) (Long et al., 2018), A DIRT-T Approach to Unsupervised
Domain Adaptation (DIRT) (Shu et al., 2018), Convolutional deep Domain
Adaptation model for Time-Series data (CoDATS) (Wilson et al., 2020), and
Adversarial Spectral Kernel Matching (AdvSKM) (Liu & Xue, 2021). The backbone
architecture of all models is a 1D-CNN network. It consists of three CNN
blocks and each block has a 1D convolutional layer, followed by 1D batch
normalization layer, ReLU activation function, 1D max pooling and Dropout. In
the first block, the kernel size of the convolutional layer is set according
to the dataset as reported in Ragab et al. (2022). After the convolutional
blocks, we apply an 1D adaptive pooling layer. All methods are trained for
$100$ epochs on all datasets. The batch size is $32$, except for Sleep-EDF,
where we use batch size of $128$. All models are trained with Adam optimizer
(Kingma & Ba, 2014) and weight decay of $10^{-4}$. Additional hyper-parameters
are reported in Table 10.

Table 6: Model backbone for the AdaTime suite. Kernel size, stride, output
channels of the convolutional layers are dataset dependent and are chosen
according to Ragab et al. (2022).

Architecture
---
| Layers
Conv Block 1 | Convolutional 1D Layer
| Batch Normalization 1D Layer
| ReLU Activation
| Max Pooling 1D Layer
| Dropout
Conv Block 2 | Convolutional 1D Layer
| Batch Normalization 1D Layer
| ReLU Activation
| Max Pooling 1D Layer
| Dropout
Conv Block 3 | Convolutional 1D Layer
| Batch Normalization 1D Layer
| ReLU Activation
| Max Pooling 1D Layer
| Dropout
| Adaptive Pooling 1D Layer
| Methods
| See Table 7 and Table 8 for details.

Table 7: Model architecture for the AdaTime dataset. Layer hyper-parameters
are dataset dependent and are chosen according to Ragab et al. (2022).

Method Architectures (Part 1)
---
| DANN
Class Output Head | Fully-connected Layer
Domain Classifier Head | Fully-connected Layer
| ReLU Activation
| Fully-connected Layer
| ReLU Activation
| Fully-connected Layer
| DeepCoral
Class Output Head | Fully-connected Layer
| DDC
Class Output Head | Fully-connected Layer
| HoMM
Class Output Head | Fully-connected Layer
| CoDATS
Class Output Head | Fully-connected Layer
| ReLU Activation
| Fully-connected Layer
| ReLU Activation
| Fully-connected Layer
| DSAN
Class Output Head | Fully-connected Layer

Table 8: Model architecture for the AdaTime dataset. Hyper-parameters are
dataset dependent and are chosen according to Ragab et al. (2022).

Method Architectures (Part 2)
---
| AdvSKM
Class Output Head | Fully-connected Layer
AdvSKM Embedder 1 | Fully-connected Layer
| Fully-connected Layer
| Batch Normalization 1D Layer
| Cosine Activation
| Fully-connected Layer
| Fully-connected Layer
| Batch Normalization 1D Layer
| Cosine Activation
AdvSKM Embedder 2 | Fully-connected Layer
| Fully-connected Layer
| Batch Normalization 1D Layer
| ReLU Activation
| Fully-connected Layer
| Fully-connected Layer
| Batch Normalization 1D Layer
| ReLU Activation
| MMDA
Class Output Head | Fully-connected Layer
| CMD
Class Output Head | Fully-connected Layer
| CDAN
Class Output Head | Fully-connected Layer
Domain Classifier Head | Fully-connected Layer
| ReLU Activation