# Enhancing Polynomial Chaos Expansion Based Surrogate Modeling using a Novel
Probabilistic Transfer Learning Strategy

Wyatt Bridgman
Sandia National Laboratories,
Livermore, California 95391, USA &Uma Balakrishnan
Sandia National Laboratories,
Livermore, California 95391, USA &Reese Jones
Sandia National Laboratories,
Livermore, California 95391, USA &Jiefu Chen
University of Houston,
Houston, Texas, 77081, USA &Xuqing Wu
University of Houston,
Houston, Texas, 77081, USA &Cosmin Safta
Sandia National Laboratories,
Livermore, California 95391, USA &Yueqin Huang
University of Houston,
Houston, Texas, 77081, USA &Mohammad Khalil
Sandia National Laboratories,
Livermore, California 95391, USA<EMAIL_ADDRESS>
###### Abstract

In the field of surrogate modeling, polynomial chaos expansion (PCE) allows
practitioners to construct inexpensive yet accurate surrogates to be used in
place of the expensive forward model simulations. For black-box simulations,
non-intrusive PCE allows the construction of these surrogates using a set of
simulation response evaluations. In this context, the PCE coefficients can be
obtained using linear regression, which is also known as point collocation or
stochastic response surfaces. Regression exhibits better scalability and can
handle noisy function evaluations in contrast to other non-intrusive
approaches, such as projection. However, since over-sampling is generally
advisable for the linear regression approach, the simulation requirements
become prohibitive for expensive forward models. We propose to leverage
transfer learning whereby knowledge gained through similar PCE surrogate
construction tasks (source domains) is transferred to a new surrogate-
construction task (target domain) which has a limited number of forward model
simulations (training data). The proposed transfer learning strategy
determines how much, if any, information to transfer using new techniques
inspired by Bayesian modeling and data assimilation. The strategy is
scrutinized using numerical investigations and applied to an engineering
problem from the oil and gas industry.

_Keywords_ Transfer Learning, Polynomial Chaos Surrogates, Bayesian
calibration, Subsurface characterization

##  1 Introduction

Obtaining cheap, accurate surrogates for complex physics models is paramount
in addressing a broad spectrum of research challenges. For example, they can
illuminate high-dimensional relationships between input parameters and model
responses [1], which makes them invaluable for model calibration [2, 3],
uncertainty quantification (UQ), optimization [4], and decision-making under
uncertainty. Hence, surrogates serve as a powerful toolset, enabling a deeper
understanding of intricate system behaviors and paving the way for robust and
informed decision-making in the face of uncertainty [5, 6, 7].

A wide variety of approaches for surrogate modeling exist and span intrusive,
physics-based methods, like reduced-order models, to data-driven neural
network surrogates [8, 9]. In the context of surrogate modeling, polynomial
chaos expansions (PCEs) [10, 11, 12] allow practitioners to construct
inexpensive yet accurate surrogates to be used in place of the expensive
forward model simulations. PCEs originated as an approach to uncertainty
quantification (UQ) based on multidimensional polynomials that are orthogonal
with respect to the joint probability distribution of the input variables.
This includes uniform (Legendre polynomials) and Gaussian (Hermite
polynomials) [13] distributions. In addition to UQ, PCEs can also be used to
build effective surrogates for models by assuming a uniform density on
parameters of interest over a domain [14]. Since the basis is complete, the
coefficients of the polynomial expansion fully characterize the model and can
be obtained using Galerkin projection [15] or regression [16]. PCEs have seen
wide-spread application in diverse fields such as engineering, physics [17,
18], finance and epidemiology [19]. For black-box simulations, non-intrusive
PCE allows the construction of surrogates using a set of simulation response
evaluations. In this context, the PCE coefficients can be obtained using
linear regression, which is also known as point collocation or stochastic
response surfaces. Regression exhibits better scalability and can handle noisy
function evaluations in contrast to other non-intrusive approaches, such as
projection. However, since over-sampling is generally advisable for the linear
regression approach, the simulation requirements become prohibitive for
expensive forward models. We propose to leverage transfer learning [20, 21],
whereby knowledge gained through similar PCE surrogate construction tasks
(source domains) is transferred to a new surrogate-construction task (target
domain) which has a limited number of forward model simulations (training
data).

Transfer learning [21, 20] has had an increasing number of applications in
machine learning (ML) [22, 23] where data-hungry, many-parameter models are
trained to capture abstract patterns in large datasets. Two important
categories of transfer learning are: _feature-based_ transfer and _parameter-
based_ transfer. Feature-based transfer assumes similarity can be found
between the source and target features using approaches based on augmentation
[24], mapping [25], and clustering [26]. In parameter-based transfer, the
focus is instead on exploiting similarity between parameters defining the
source and target task models [27, 28, 29] with methods like parameter sharing
and regularization. Here, we focus on parameter-based transfer since the
assumption that the input-to-output map is similar between tasks has a variety
of applications in physics and engineering, as we will illustrate in Sec.
3.3.

Parameter-based transfer learning techniques are often deterministic in that
they yield a point estimate for the target task model parameters and so do not
offer a strategy to quantify uncertainty. In this work, we propose a
probabilistic, parameter-based transfer learning strategy to address this gap.
The approach consists of transferring knowledge between Bayesian inverse
problems where the source knowledge is represented as a prior over the target
task which is informed by the posterior over the source task. A key novelty of
this methodology is controlling the transfer of by tempering the source
posterior to avoid negative transfer, whereby accuracy is impeded by the
source information. This is carried out by posing an optimization problem to
determine the optimal amount of knowledge to carry over from the source task.

The remainder of the paper is organized as follows. The next section, Sec. 2,
describes the probabilistic transfer learning methodology where an
optimization and tempering procedure are introduced to control knowledge
transfer. Then Sec. 3 presents the result of three demonstrations. First,
domain adaptation of simple polynomial models is described in Sec. 3.1 and
then extended to a more complex model in Sec. 3.2. The proposed workflow is
then applied to a subsurface scattering physics surrogate in Sec. 3.3.
Finally, a summary and a perspective on future work is given in Sec. 4.

##  2 Methodology

In this section, we describe the main methodological concepts used in the
proposed transfer learning framework. We first describe surrogate modeling and
calibration with PCEs using standard Bayesian linear regression. Next,
transfer learning in the context of Bayesian inverse problems is introduced
along with the key concept of tempering to control the amount of knowledge
transferred from source to target. In this work tempering takes the form of
parametric transformations applied to distributions to diffuse probability
mass and thereby control how much knowledge is transferred. Different
approaches to determine optimal tempering parameters are discussed, and
several choices for the objective function motivated and described in detail.

###  2.1 PCE Surrogate Modeling

Polynomial Chaos expansions have a long history in computational science and
engineering [10, 30, 11, 31, 32, 33]; here we review the fundamentals in the
context of surrogate modeling.

Assume that a quantity of interest, $y$, of a physical system is modeled by

$y=\mathcal{M}\left(\mathbf{x}\right)\ ,$ (1)

in which $\mathbf{x}=\left\\{x_{1},\ldots,x_{n}\right\\}\in\mathbb{R}^{n}$ is
a vector of model parameters. Assuming, without loss of generality, that
$x_{1},\ldots,x_{n}$, through some (usually linear) transformation of the
corresponding model parameters vary within the $n$-dimensional hypercube
$\left[-1,1\right]^{n}$, one can construct a polynomial approximation of $y$,
as

$\displaystyle y=P\left(\mathbf{x}\right)$
$\displaystyle=\theta_{0}+\sum_{j=1}^{n}\theta_{j}\psi_{j}\left(x_{j}\right)+\sum_{j=1}^{n}\sum_{k=1}^{j}\theta_{jk}\psi_{jk}\left(x_{j},x_{k}\right)$
$\displaystyle\ \
+\sum_{j=1}^{n}\sum_{k=1}^{j}\sum_{h=1}^{k}\theta_{jkh}\psi_{jkh}\left(x_{j},x_{k},x_{h}\right)+\cdots\
,$ (2)

where $\theta_{i\ldots}$ are coefficients and $\psi_{i\ldots}$ are, in the
case of polynomial chaos expansions (PCE), multivariate polynomial basis
functions. The PCE basis is orthogonal with respect to the joint uniform
probability density function (PDF) of $\mathbf{x}$ on $\left[-1,1\right]^{n}$.
In practice, this PCE is truncated at a finite number of basis functions. For
a total order truncation, the number of terms, $N_{\textrm{PCE}}$, in the
expansion is given by

$N_{\textrm{PCE}}=\frac{\left(n+d\right)!}{n!\ d!}\ ,$ (3)

where $d$ is the degree of the expansion. The coefficient vector,
${\bm{\theta}}\in\mathbb{R}^{N_{\textrm{PCE}}}$, can be determined via linear
regression using a training data set:

$\displaystyle X$
$\displaystyle=\left\\{\mathbf{x}^{1},\ldots,\mathbf{x}^{N_{p}}\right\\}$ (4)
$\displaystyle Y$ $\displaystyle=\left\\{y^{1},\ldots,y^{N_{p}}\right\\}\ ,$
(5)

Generally, it is advisable to have $N_{p}>N_{\textrm{PCE}}$, resulting in an
over-determined system of equations for ${\bm{\theta}}$. Since the truncated
PCE, $P_{d}\left(x\right)$, is an approximations of $y$, we will model each
sample of the data as

$y^{k}=P_{d}\left(\mathbf{x}^{k},{\bm{\theta}}\right)+\epsilon^{k}\ ,$ (6)

and assume that the error term $\epsilon^{k}$ is independently and identically
distributed white Gaussian noise, i.e.
$\epsilon^{k}\sim\mathcal{N}(0,\gamma^{2})$. In this case, the likelihood
function for the coefficients, ${\bm{\theta}}$, has the form:

$p\left(D|{\bm{\theta}}\right)=\mathcal{N}(\bar{{\bm{\theta}}}_{\rm
lik},\bm{\Sigma}_{\rm lik})\ ,$ (7)

with $D=\left\\{X,Y\right\\}$,

$\displaystyle\bar{{\bm{\theta}}}_{\rm lik}$
$\displaystyle=\left(\mathbf{A}^{T}\mathbf{A}\right)^{-1}\mathbf{A}^{T}\,\mathbf{Y}$
(8) $\displaystyle\bm{\Sigma}_{\rm lik}$
$\displaystyle=\gamma^{2}\left(\mathbf{A}^{T}\mathbf{A}\right)^{-1}\ ,$ (9)

and $\mathbf{A}$ is the so-called Vandermonde matrix of polynomial basis
functions evaluated at the set of parameter vector realizations $X$.

Assuming, for reasons of conjugacy, that the prior PDF of ${\bm{\theta}}$ is
also Gaussian,

$p\left({\bm{\theta}}\right)=\mathcal{N}(\bar{{\bm{\theta}}}_{\rm
pr},\bm{\Sigma}_{\rm pr})\ ,$ (10)

the posterior PDF, given by Bayes’ law, is also Gaussian of the form:

$\displaystyle p\left({\bm{\theta}}|D\right)$
$\displaystyle=\frac{p\left(D|{\bm{\theta}}\right)\
p\left({\bm{\theta}}\right)}{p\left(D\right)}$
$\displaystyle=\mathcal{N}(\bar{{\bm{\theta}}}_{\rm post},\bm{\Sigma}_{\rm
post})\ ,$ (11)

with mean $\bar{{\bm{\theta}}}_{\rm post}$ and covariance $\bm{\Sigma}_{\rm
post}$:

$\displaystyle\bar{{\bm{\theta}}}_{\rm post}$
$\displaystyle=\left(\left(\bm{\Sigma}_{\rm
pr}\right)^{-1}+\left(\bm{\Sigma}_{\rm
lik}\right)^{-1}\right)^{-1}\left(\left(\bm{\Sigma}_{\rm
pr}\right)^{-1}\bar{{\bm{\theta}}}_{\rm pr}+\left(\bm{\Sigma}_{\rm
lik}\right)^{-1}\bar{{\bm{\theta}}}_{\rm lik}\right)$ (12)
$\displaystyle\bm{\Sigma}_{\rm post}$
$\displaystyle=\left(\left(\bm{\Sigma}_{\rm
pr}\right)^{-1}+\left(\bm{\Sigma}_{\rm lik}\right)^{-1}\right)^{-1}\ .$ (13)

###  2.2 Transfer Learning

Limited available data can be used to partially inform the PCE coefficients
through the analysis of the likelihood function in Eq. (7). Bayes’ law, as
shown in the previous section, provides the mechanism of further informing the
PCE coefficients using knowledge gained prior to the analysis of the target
data. In a transfer learning setting, this prior knowledge is gained through
the training of the PCE surrogate on other (possibly similar) data sets.
However, the Bayes’ update _should not_ be applied as is in this scenario
since the level of similarity of those other sets of data relative to the
current one is not known a priori.

In this case, we propose the following extension to the Bayes’ update to merge
current knowledge gained, as captured by the likelihood function, and prior
knowledge as encoded in the prior PDF:

$p\left({\bm{\theta}}|D,\beta\right)\propto p\left(D|{\bm{\theta}}\right)\
p\left({\bm{\theta}}\right)^{\beta}\ ,$ (14)

The tempering parameter $0\leq\beta\leq 1$ allows the formalism to diffuse the
prior knowledge, with the limiting cases of $\beta=0$ eliciting no use of
prior knowledge and $\beta=1$ reverting to the traditional Bayes’ law. This
specific form of tempering via exponentiation diffuses bell-shaped priors by
widening their extent, thus making the transferred information less certain;
it is related to the _power prior_ [34, 35] employed outside of transfer
learning. This added flexibility allows us to control how much knowledge in
the PCE coefficients is transferred from the prior PDF to the posterior, with
diminishing effect as $\beta$ approaches 0.

In this formalism, the single tempering parameter $\beta$ controls the amount
of knowledge to be transferred from the source training task, with data
$D_{S}$, to the target training task involving the construction of PCE
surrogates, with target data $D_{T}$. The target data $D_{T}$ is used in
formulating the likelihood function, and the source data $D_{S}$ is used in a
separate calibration task to provide a prior PDF on ${\bm{\theta}}$.

We can determine the tempering parameter $\beta$ using one of the following
approaches:

  * •

_Hierarchical Bayes_ : One can proceed with the joint inference of the PCE
coefficients ${\bm{\theta}}$ and tempering parameter $\beta$ by formulating
the joint posterior PDF:

$\displaystyle p\left({\bm{\theta}},\beta|D\right)$ $\displaystyle\propto
p\left({\bm{\theta}}|D,\beta\right)\ p\left(\beta\right)\ ,$ (15)

with $p\left(\beta\right)$ being the prior PDF on the tempering parameter
$\beta$, for example a uniform PDF on [0,1].

  * •

_Empirical Bayes_ : Utilize a point estimate for $\beta$ that may be obtained
through a separate optimization problem. The associated objective function may
be based on cross-validation or maximum expected data-fit, for example.

For this investigation, we will focus on the empirical Bayesian approach and
formulate an optimization problem to obtain a point estimate $\beta^{*}$ for
$\beta$, the tempering parameter.

###  2.3 Tempering

_Tempering_ is, in general, a means of reducing the certainty of previous
information in the form of a mollified prior. This usually involves changing
the covariance structure of the target prior (source posterior) in a
parametric way, such as convolving it with a kernel. Here we use power
tempering and focus on Gaussian priors.

To define tempering, let $\pi_{S}({\bm{\theta}}|\beta)$ denote the likelihood
(or posterior) over parameters ${\bm{\theta}}$ for the source task. Here,
$\pi_{S}$ is also a function of the tempering parameter $\beta$. This provides
a compact notation for indicating that a $\beta$-dependent tempering
transformation is applied to the original, unmodified source likelihood
$\pi_{S}({\bm{\theta}})$ to produce a tempered distribution
$\pi_{S}({\bm{\theta}}|\beta)$. By taking the tempered source likelihood to be
the prior for the target calibration, the posterior distribution for the
target task is $\beta$-dependent and given by

$\pi_{p}({\bm{\theta}}|\beta)\propto\pi_{S}({\bm{\theta}}|\beta)\pi_{T}({\bm{\theta}})$
(16)

where $\pi_{T}({\bm{\theta}})$ is the target likelihood.

In this work, we use focus on power tempering, a transformation defined by
raising the distribution to a power $\beta$

$\pi_{S}({\bm{\theta}}|\beta)\propto\pi_{S}({\bm{\theta}})^{\beta}$ (17)

In the case of Gaussian distributions, the tempered source likelihood has the
form
$\pi_{S}({\bm{\theta}}|\beta)=\mathcal{N}({\bm{\theta}};{\bm{\mu}}_{S},(1/\beta)\bm{\Sigma}_{S})$,
as raising a Gaussian to a power $\beta$ simply scales the covariance matrix.
Letting

$\displaystyle\pi_{T}({\bm{\theta}})$ $\displaystyle=$
$\displaystyle\mathcal{N}({\bm{\theta}};{\bm{\mu}}_{T},\bm{\Sigma}_{T}),$ (18)
$\displaystyle\pi_{S}({\bm{\theta}}|\beta)$ $\displaystyle=$
$\displaystyle\mathcal{N}({\bm{\theta}};{\bm{\mu}}_{S},(1/\beta)\bm{\Sigma}_{S}),$
(19)

the resulting tempered Gaussian posterior is defined by the first two moments

$\displaystyle\bm{\Sigma}^{-1}_{p}(\beta)$ $\displaystyle=$
$\displaystyle\bm{\Sigma}_{T}^{-1}+\beta\bm{\Sigma}_{S}^{-1}$ (20)
$\displaystyle{\bm{\mu}}_{p}(\beta)$ $\displaystyle=$
$\displaystyle\bm{\Sigma}_{p}(\beta)\left[\bm{\Sigma}_{T}^{-1}{\bm{\mu}}_{T}+\beta\bm{\Sigma}_{S}^{-1}{\bm{\mu}}_{S}\right]$
(21)

Note that for Gaussian distributions, power tempering preserves the mean of
the source likelihood while uniformly scaling the covariance. This scaling
preserves the correlation coefficient between variables and so maintains the
same correlation structure while diffusing knowledge by decreasing the overall
uncertainty as $\beta$ goes to zero. This can be seen by noting that the
correlation matrix $\mathbf{R}$ (matrix of correlation coefficients) can be
extracted from the covariance matrix $\bm{\Sigma}$ as
$\mathbf{R}=\mathbf{D}^{-1}\bm{\Sigma}\mathbf{D}^{-1}$ where
$\mathbf{D}=\sqrt{{\rm diag}(\bm{\Sigma})}$. Hence, it can be easily shown
that the correlation coefficients remain unchanged when scaling the covariance
matrix by $\beta$ as is done in power tempering. While we focus on power
tempering in this work, note that other tempering transformations, such as
ones based on convolution, could be adopted in the proposed work flow
resulting in different mechanisms of diffusing the knowledge encoded in the
likelihood PDFs. The optimal choice of tempering transformation might be
dependent on the data distribution and the models being calibrated.

###  2.4 Objective functions

In this section, we describe four objective functions considered for the
empirical Bayes formulation of transfer learning presented in Sec. 2.2. These
objectives depend on the tempering parameter $\beta$ through the source
likelihood and/or target posterior. The objective functions are chosen to be
sensitive to common features between the source and target likelihoods and
behave in a qualitatively similar fashion. The resulting optimal tempering
parameter $\beta^{*}$ determines how much knowledge from the source task
should be diffused in the process of transferring information to the target
task.

The first objective is based on the Expected Data Fit (EDF), an information-
theoretic quantity defined by $\mathbb{E}_{\pi_{p}}\left[\log\pi_{T}\right]$
that effectively measures the average data fit provided by the posterior
distribution $\pi_{p}$ over parameters. It is commonly used in Bayesian
inference and statistical modeling to measure how well a probabilistic model
fits observed data. It quantifies the likelihood of observing the given data
under the model [36]. Hence, by maximizing the EDF, we improve the target data
fit which leads to the following objective function and equivalent
minimization problem

$\operatorname{argmin}_{\beta}-\mathbb{E}_{\pi_{p}({\bm{\theta}}|\beta)}\left[\log\pi_{T}({\bm{\theta}})\right]$
(22)

This problem can be expressed in several different equivalent forms including:

$\displaystyle\operatorname{argmin}_{\beta}-\mathbb{E}_{\pi_{p}({\bm{\theta}}|\beta)}\left[\log\pi_{T}({\bm{\theta}})\right]$
$\displaystyle=\operatorname{argmin}_{\beta}C(\pi_{p}({\bm{\theta}}|\beta),\pi_{T}({\bm{\theta}}))$
(23)
$\displaystyle=\operatorname{argmin}_{\beta}D_{\text{KL}}(\pi_{p}({\bm{\theta}}|\beta)\;\|\;\pi_{T}({\bm{\theta}}))+H(\pi_{p}({\bm{\theta}}|\beta))$
$\displaystyle=\operatorname{argmin}_{\beta}D_{\text{KL}}(\pi_{p}({\bm{\theta}}|\beta)\;\|\;\pi_{S}({\bm{\theta}}|\beta))-e(\beta)$

where $C$ denotes the cross-entropy, $D_{\text{KL}}$ the Kullback–Leibler
divergence, $H$ the entropy, and
$e=\int\pi_{T}({\bm{\theta}})\pi_{T}^{0}({\bm{\theta}}|\beta)\mathrm{d}{\bm{\theta}}$
the model evidence. For example, minimizing the negative EDF is seen to be
equivalent to minimize the cross entropy between the tempered posterior
$\pi_{p}({\bm{\theta}}|\beta)$ and target likelihood $\pi_{T}({\bm{\theta}})$,
a common objective function in machine learning applications.

In the case where all distributions are Gaussian, the EDF takes the form

$\begin{split}\operatorname{argmin}_{\beta}\frac{1}{2}\bigl{[}({\bm{\mu}}_{p}(\beta)-{\bm{\mu}}_{T})^{T}\bm{\Sigma}_{T}^{-1}({\bm{\mu}}_{p}(\beta)-{\bm{\mu}}_{T})\\\
+\operatorname{tr}\left(\bm{\Sigma}_{T}^{-1}\bm{\Sigma}_{p}(\beta)\right)+\log\operatorname{det}(2\pi\bm{\Sigma}_{T})\bigr{]}\end{split}$
(24)

The second objective is based on the KL-divergence which also originates from
information theory and statistics [37]. It quantifies the difference between
two probability distributions and is widely used in various fields, including
machine learning, to measure the dissimilarity between two probability
distributions. It plays a pivotal role in tasks such as model comparison,
optimization, and information retrieval. By taking the KL-divergence term from
Eq. (23), we arrive at a second objective function

$\operatorname{argmin}_{\beta}D_{\text{KL}}(\pi_{p}({\bm{\theta}}|\beta)\;\|\;\pi_{S}({\bm{\theta}}|\beta))$
(25)

which we denote as KLD.

For Gaussian distributions, the optimization has the form

$\begin{split}\operatorname{argmin}_{\beta}\frac{1}{2}\bigl{[}({\bm{\mu}}_{p}(\beta)-{\bm{\mu}}_{S})^{T}\beta\bm{\Sigma}_{S}(\beta)^{-1}({\bm{\mu}}_{p}(\beta)-{\bm{\mu}}_{S})\\\
+\operatorname{tr}\left(\beta\bm{\Sigma}_{S}^{-1}\bm{\Sigma}_{p}(\beta)\right)-\log\frac{\operatorname{det}(\bm{\Sigma}_{p}(\beta))}{\operatorname{det}((1/\beta)\bm{\Sigma}_{S}(\beta))}-d\bigr{]}\end{split}$
(26)

Recall that $d$ is the degree of the PCE model.

The third objective is based on the model evidence
$e=\int\pi_{T}\pi_{S}(\beta)\operatorname{d}\\!{{\bm{\theta}}}$ which forms
the posterior normalizing factor of the distribution
$\pi_{S}(\beta)\pi_{T}\propto\pi_{p}(\beta)$. It also known as the marginal
likelihood, in Bayesian statistics. In Bayesian inference, the model evidence
provides the probability of the observed data marginalized over all possible
model parameters ${\bm{\theta}}$, in essence giving the likelihood that a
particular model resulted in the data [38]. Hence, this quantity is often used
in model selection, a Bayesian strategy where the most appropriate model is
selected from a family of possible models. By maximizing this quantity, we
arrive at our third objective function

$\operatorname*{arg\,max}_{\beta}\int\pi_{T}({\bm{\theta}})\pi_{S}({\bm{\theta}}|\beta)\operatorname{d}\\!{{\bm{\theta}}}$
(27)

which we denote by ME.

For Gaussian distributions, this reduces to

$\operatorname*{arg\,max}_{\beta}\hskip
2.84526pt\mathcal{N}({\bm{\mu}}_{T};{\bm{\mu}}_{S},\bm{\Sigma}_{T}+(1/\beta)\bm{\Sigma}_{S})$
(28)

The fourth and final objective, Dice Similarity (DS) is widely used in image
segmentation and pattern recognition [39] to measure the similarity between
two sets or regions. It has also been used to compare probability
distributions [40] where it can be seen as a normalized inner product that
reflects how much the two distributions overlap. The DS satisfies $0\leq
D_{\text{dice}}(p,q)\leq 1$ for distributions $p,q$ with
$D_{\text{dice}}(p,q)=1$ if and only if $p=q$. Hence, this provides an
interpretable, normalized notion of similarity between two distributions that
can be use to formulate the objective

$\operatorname*{arg\,max}_{\beta}D_{\text{dice}}(\pi_{S}({\bm{\theta}}|\beta),\pi_{T}({\bm{\theta}}))$
(29)

where we omit the lengthy Gaussian formula for the sake of brevity.

Note that objective function from each of these four optimization problems can
be expressed as
$\int\pi_{T}({\bm{\theta}})\pi_{S}({\bm{\theta}}|\beta)w({\bm{\theta}}|\beta)\operatorname{d}\\!{{\bm{\theta}}}$
where $w$ is a, possibly $\beta$-dependent, weighting function. Namely, each
is a weighted inner product between the target and source likelihoods and
measures some form of overlap between the two distributions. These inner
products are also associated with norms if the two functions are normalized.

##  3 Results

In this section we first explore the behavior of four objective functions
introduced in Sec. 2 using easily interpretable test problems. Sec. 3.1
details an investigation of transfer learning for _domain_ adaptation across
all four objective functions. We compare the performance of the optimal
transfer evoked by each objective to no transfer and full transfer for four
domain overlaps with three different polynomial models. This exhaustive study
is used to down-select from the set of objective functions to a single, best-
performing one. Subsequently, in Sec. 3.2, we apply the proposed method to a
_task_ adaptation problem using a more complex generative model to validate
previous observations and show that they hold for task adaptation. Finally, in
Sec. 3.3, the transfer learning framework is deployed on a problem of
pragmatic interest involving PCE surrogates calibrated to models of subsurface
scattering. Here, again we compare to no and full transfer to demonstrate the
method outperforms each.

###  3.1 Domain adaptation

Domain adaptation addresses the challenge of training models on source data
and generalizing their performance to target data with different
distributions. While traditional calibration assumes that source and target
data come from the same distribution, real-world scenarios often involve
different feature spaces or evolving statistical properties. Collecting target
data that perfectly matches the source distribution can be challenging, and
utilizing related annotated data is often preferred to creating new labeled
datasets. Domain adaptation minimizes distribution differences between the
source and target domains to enable effective knowledge transfer.

In each of the examples in this section, we utilize the pushed-forward
posterior distribution (PFP) to evaluate the target task model after transfer
learning. The PFP leverages a parameter-to-prediction mapping that establishes
a connection between parameters and the observed outcomes [41]. The PFP is the
pushforward density [42, 43] of the posterior distribution (14) over model
parameters ${\bm{\theta}}$ through the model function $f(x;{\bm{\theta}})$ at
unobserved data $x=x^{i}$. In the case where the tempered posterior is
Gaussian
$\pi_{p}({\bm{\theta}}|\beta)=\mathcal{N}({\bm{\theta}}|{\bm{\mu}}_{p}(\beta),\bm{\Sigma}_{p}(\beta))$,
the PFP density is defined by

$\operatorname{PFP}(\hat{y}^{i}|\beta)=\mathcal{N}(\hat{y}^{i}|{\bm{\mu}}(\beta),\bm{\Sigma}(\beta))$
(30)

where ${\bm{\mu}}=\mathbf{A}(x^{i}){\bm{\mu}}_{p}(\beta)$ and
$\mathbf{A}(x^{i})\bm{\Sigma}_{p}(\beta)\mathbf{A}^{T}(x^{i})$, which follows
from the standard results on linear transformations of a Gaussian random
variables [44] for model that is linear in its parameters such as PCE. Recall
$\mathbf{A}$ is the Vandermonde matrix. By computing the log probability of
the PFP across data $(x_{i},\hat{y}_{i})$, $i=1,\ldots,m$ as a function of
$\beta$

$\operatorname{LPFP}(\beta)=\sum_{i=1}^{m}\log\operatorname{PFP}(\hat{y}^{i}|\beta)$
(31)

we obtain a metric that helps in assessing the model quality, aggregating
uncertainties effectively, and making informed decisions based on model
performance on unobserved data.

####  3.1.1 Comparison of objective functions

In the first demonstration, we aim to transfer information between the source
and target domains by leveraging the knowledge learned from source domain to
improve the performance of a model in the target domain when there is a shift
or mismatch between the two domains. An good objective function will perform
better than no or full transfer of knowledge and avoid negative learning
relative to these two baselines.

For this demonstraton, we focus on the construction of PCE-based surrogates
for a (ground truth) one-dimensional third-order polynomial:

$y=0.3((1/3)x^{3}-2.25x).$ (32)

The calibration problem is constructing a Legendre polynomial-based surrogate
for the source domain $x\in[-0.2,0.3]$ and transferring the learned knowledge
to various target domains of length $0.4$ in the interval $[-3.0,3.0]$. For
each test the interval encompassing the source and target domains is scaled
and shifted using linear transformations to fit within the range $[-1,1]$. We
approximate the data in the source and target domains using a truncated PCE:

$\displaystyle\hat{y}^{i}=\hat{y}(x^{i})\approxeq
f(x^{i};{\bm{\theta}})=\sum_{j=0}^{p}\theta_{j}\psi_{j}(x^{i}),$ (33)

where $\psi_{j}(x^{i})$ is the orthonormal Legendre polynomial of degree
$d\in\\{1,2,3\\}$ and ${\bm{\theta}}$ is the vector of model parameters. The
PCE of degree $d$ has $p=d+1$ parameters in this 1D case. To mimic noisy
observations, we add random Gaussian white noise as

$\hat{y}^{i}=f(x^{i};{\bm{\theta}})+\epsilon_{i}.$ (34)

and uniformly sample 16 data points from source domain and 4 data points from
chosen target domain. We form the $\mathbf{A}_{s}\in\mathbb{R}^{16\times p}$
and $\mathbf{A}_{t}\in\mathbb{R}^{4\times p}$ Vandermonde matrices of
orthonormal Legendre polynomial basis functions to calculate the prior and
likelihood PDFs. The optimal tempering parameter $\beta^{*}$ is obtained using
objective functions EDF, KLD, ME, and DS from Sec. 2.4.

Figure 1: Optimal tempering parameter $\beta^{*}$ for various objective
functions and model types across different target domains.

Fig. 1 shows box plots of an ensemble of 100 realizations which illustrate
varying levels of knowledge transfer across different objective functions and
model types. In the figure, the boxes represents the interquartile range
($IQR$), showing the middle 50% of the data, while the line within the box
indicates the median. In Fig. 1, the $x$-axis indicates the distance between
the mean of source and the mean of target domains, while the $y$-axis
represents the optimal amount of knowledge transferred from the source to the
target domain, as measured by the tempering exponent $\beta$. The value of
$\beta$ ranges from 0 (indicating no transfer of knowledge) to 1 (representing
full transfer), with intermediate values reflecting partial knowledge
transfer.

The figure reveals distinct patterns for different objective functions and
model types. Specifically, for quadratic and cubic models, the objective
functions KLD and ME facilitate complete knowledge transfer from the source to
the target domain. However, for the linear model, the knowledge transfer is
partial, especially when the target domain is far from the source domain,
which appears to be optimal given the dissimilarity of the tasks. On the
contrary, when the target domain coincides or stays close to the source
domain, the linear model transfers almost full knowledge. In contrast, when
using the DS objective function, the transfer of information is minimal for
quadratic and cubic models. For the linear model, it transfers more knowledge
when the source and target domains coincide, but this knowledge transfer
diminishes as the target domain moves away from the source domain. Lastly, the
EDF objective function results in complete knowledge transfer from the source
to the target domain when the model is cubic. For quadratic and linear models,
complete knowledge transfer occurs when the source and target domains
coincide, intersect, or remain close to each other, but no knowledge transfer
takes place when the target domain moves away from the source domain. The
results portrayed in Fig. 1 demonstrate that the EDF objective function
optimizes knowledge transfer appropriately and effectively between the source
and target domains across different models types and domain configurations.

Figure 2: Performance gain in the proposed transfer learning against the no
transfer baseline for various objective functions, model types and target
domains; $y$-axis = performance gain against no transfer =
$\operatorname{LPFP}(\beta^{*})-\operatorname{LPFP}(0)$.  Figure 3:
Performance gain in the proposed transfer learning against the full transfer
baseline for various objective functions, model types and target domains;
$y$-axis = performance gain against full transfer =
$\operatorname{LPFP}(\beta^{*})-\operatorname{LPFP}(1)$.

Next we calculate the LPFP for the same models and objectives . Figures 2 and
3 present a comparison of the LPFP scores for optimal transfer
$\beta=\beta^{*}$ relative to no transfer $\beta=0$ and to full transfer
$\beta=1$ , respectively. The $x$-axis in each panel is the domain shift, and
the $y$-axis is the difference in LPFP. Where the difference is positive this
indices the optimal transfer, as defined by the selected objective, has
outperformed the respective baseline. Across all the scenarios, it appears
that only EDF generally avoids negative transfer and has improvements over
both baselines. The KLD objective has some regions of comparable performance
but also significant negative learning. It is also noteworthy that the
performance relative to no transfer is typically the complement of that
relative to full transfer.

Figure 2 illustrates that regardless of the chosen objective function, the
$\beta^{*}$ consistently outperforms the no transfer approach for the cubic
model. Similarly, for the quadratic and linear models, $\beta^{*}$ exhibits
superior performance as long as the target domain remains close to the source
domain, irrespective of the objective function used. However, when the target
domain moves significantly away from the source domain, the performance of
$\beta^{*}$ diminishes for the quadratic model, especially for KLD and ME
objectives. Additionally, for the linear model, no transfer surpasses the
performance of $\beta^{*}$ by a substantial margin for KLD, ME, and DS
objective functions, whereas $\beta^{*}$ performs similarly to no transfer for
the EDF objective.

Figure 3 illustrates that in the case of the cubic model, both the $\beta^{*}$
and $\beta=1$ yield comparable results for the EDF, KLD, and ME objective
functions. However, for the DS objective function, when the target domain
intersects, merges, or remains closely aligned with the source domain, full
transfer outperforms optimal transfer. Moving on to the quadratic model, once
again, the scenarios where $\beta^{*}$ and $\beta=1$ perform equally are
observed with the KLD and ME objective functions. Nevertheless, when the
target domain deviates significantly from the source domain, $\beta^{*}$
exhibits superior performance for the EDF and DS objective functions. For the
linear model, an interesting pattern emerges. In cases where the target domain
lacks intersection with the source domain, $\beta^{*}$ significantly
outperforms full transfer ($\beta=1$) in the context of the EDF objective
function. However, across the KLD, ME, and DS objective functions, $\beta^{*}$
consistently outperforms $\beta=1$ when the target domain moves far away from
the source domain.

It is important to emphasize that the true underlying model is cubic, as
represented by Eq. (32). When approximating this truth using truncated PCE,
whether through a linear or quadratic model (Eq. (33) with $d=1$ or $2$), no
knowledge transfer is expected when there is insignificant similarity between
the trends in the target and source domains. On the other hand, transfer,
whether partial or full, occurs when an intersection between the two domains
exists, and the extent of transfer is contingent upon the degree of
intersection. Figures 1, 2 and 3 provide clear evidence that the EDF objective
function consistently outperforms other objective functions (KLD, ME, DS)
across all model types (linear, quadratic, and cubic) when used with the
$\beta^{*}$ strategy. These findings underscore the efficacy of the
$\beta^{*}$ strategy and the prominence of the EDF objective function in the
context of domain adaptation scenarios. Hence, for the further analysis in
this paper, we consider only EDF objective function.

####  3.1.2 Illustration of performance of the EDF objective

Now we demonstrate the performance of EDF objective function for a given model
type, target domain, and three different $\beta$ values, namely, $\beta=0$ (no
transfer), $\beta=1$ (full transfer) and $\beta=\beta^{*}$ (optimum transfer).
Figure 4 shows the underlying truth given by Eq. (32), source domain and four
different target domains denoted by A (target domain is far from the source
domain), B (target domain is close to the source domain), C (target domain
intersects partially with the source domain), and D (target domain merges with
the source domain). In order to understand the performance of all models
(linear, quadratic and cubic) over various target domains in transfer of
knowledge, we calculate the PFP to obtain a probabilistic predictions for each
target domain (A, B, C, and D). We randomly generate 16 data points from
source domain and 4 data points from target domain for 100 different
realizations using Latin Hypercube Sampling.

Figure 4: Representation of truth, source and target domains.

Figure 5 shows the PFP of the linear model, where the source domain remains
constant, while varying target domains are considered. Each plot includes a
95% confidence interval derived from 100 instances. For target domain A, it is
notable that the absence of knowledge transfer yields a closer fit to the true
model compared to optimal knowledge transfer and full knowledge transfer.
However, the trend shifts for target domains B, C, and D. In these cases,
optimal knowledge transfer and full knowledge transfer provide a markedly
better fit to the underlying truth while exhibiting reduced uncertainty, in
contrast to the case of no knowledge transfer. Applying this same analysis to
a quadratic model fitted to data gave similar treends, which have been omitted
for brevity.

Figure 5: PFP for the linear model with fixed source domain and four different
target domains for EDF objective. Red: source data, blue: target data, green:
model fits.

Fig. 6 depicts the cubic model fits, adhering to a consistent source domain
while experimenting with diverse target domains. Similar to the previous
models, each plot is accompanied by a 95% confidence interval derived from 100
instances. Unlike the linear and quadratic models, in the case of the cubic
model, optimal knowledge transfer and full knowledge transfer distinctly
outperform the absence of knowledge transfer. This improved performance holds
true across all target domains (A-D), with both optimal and full knowledge
transfer demonstrating a superior alignment with the underlying truth while
exhibiting reduced uncertainty.

Following a comprehensive investigation into transfer learning for domain
adaptation encompassing four distinct objective functions across three models
(linear, quadratic and cubic), our findings indicate that: (a) the EDF
objective function optimizes knowledge transfer effectively between the source
and target domains across different models types and domain configurations;
(b) the other objective functions (KLD, ME & DS) exhibit similar performance
to EDF across various domains, only when the model is generative; (c) the
performance gain in the proposed transfer learning ($\beta=\beta^{*}$) with
the EDF objective outperforms no transfer ($\beta=0$) and full transfer
($\beta=1$) across various domains and models considered. Consequently, we
have elected to adopt the EDF objective function as our primary approach for
illustrating task adaptation in the following section.

Figure 6: PFP for the cubic model with fixed source domain and four different
target domains for EDF objective. Red: source data, blue: target data, green:
model fits.

###  3.2 Task adaptation with the Ishigami function

In contrast to domain adaptation, which we explored in the previous section,
task adaptation considers the case where the source and target data come from
the same distributions but the forward models, or tasks, themselves are
different. We assume prior knowledge that the two tasks, while not exactly the
same, share similarities that can be exploited to transfer knowledge from
source to target.

In addition, we would like to illustrate how the proposed method performs on a
higher-dimensional generative model that is not polynomial. Hence, we
construct a calibration task based on the Ishigami function [45] defined as
$f(x_{1},x_{2},x_{3};{\bm{\theta}})=\sin(x_{1})+a\sin^{2}(x_{2})+bx_{3}^{4}\sin(x_{1})$,
where ${\bm{\theta}}=(a,b)$. The Ishigami function exhibits strong
nonlinearity and nonmonotonicity as well as complex dependence on $x_{3}$
[46]. For simplicity, we take $a=0$, $b=1$ so that we have a function of two
variables $x_{1}$, $x_{3}$ which we denote as $f(x,y)$. We introduce
parametric dependence on a single parameter $\theta$ by translating $x$, i.e.,
$f(x,y;\theta)=f(x-\theta,y)$. The source and target domains are then fixed to
be $[-1,1]^{2}$ while different tasks are obtained by varying $\theta$. Hence,
the generative model is given by

$f(x,y;\theta)=\sin(x-\theta)+y^{4}\sin(x)$ (35)

We let $p=||\theta_{T}-\theta_{S}||$ represent the difference between the
$\theta_{T}$, $\theta_{S}$, the source and target values of $\theta$,
respectively and display $f(x,y;\theta)$ at the minimum and maximum values of
$p$ in Fig. 7.

Figure 7: Ishigami function at minimum and maximum $p$ values.

The PCE model has the form of a tensor-product polynomial basis over
$\mathbb{R}^{2}$ using the Legendre polynomials with maximum degree $d=3$.
Here, the number of PCE coefficients gives ${\bm{\theta}}\in\mathbb{R}^{10}$.
To carry out transfer learning, we focus on the EDF objective function (22) to
determine $\beta$, as it showed superior behavior in domain adaptation in
Sec. 3.1. By looking at the optimal $\beta$ values and performance metrics
such as pushforward error, we can examine the performance of transfer learning
in an analogous manner to Sec. 3.1.

Figure 8: Ishigami problem: RMSE of target model evaluated with the mean
parameters of the PFP distribution. The cases of no transfer, optimal
transfer, and full transfer are displayed in red, blue, and green,
respectively. Optimal $\beta$ is shown in dashed black. Each quantity is
plotted as a function of $p$. The confidence bounds reflect the variance from
random samplings of the source and target domains and are drawn at 1 standard
deviation.

To measure the predictive performance of the source and target PCE surrogate
models, a held-out set of $N_{\text{val}}=1000$ source and target data was
used to define a validation root mean squared error (RMSE) for the source and
target models using the mean parameters from the PFP distributions. This
approach is motivated by the ability of surrogate models, whether PCE or
others, to exactly fit or overfit datasets that are sparse with respect to the
number of model parameters. Hence, we expect the target model to fit the data
exactly so that the RMSE over the data used to calibrate the model is
uninformative. To account for the effect of sampling on the transfer learning
methodology, we average the RMSE performance over $N=500$ trials where data is
randomly sampled from the source and target domains. The number of source data
points is $N_{S}=40$ while the number of target data points is $N_{T}=11$.
This is just sufficient to obtain an over-determined least squares problem for
the ${\bm{\theta}}$.

Fig. 8 shows the mean validation RMSE and confidence intervals for optimal
transfer as well as optimal $\beta$ values as a function of the difference $p$
between the source and target values for $t$. Also displayed are the cases of
no transfer, $\beta=0$, and full transfer $\beta=1$ to provide baseline
comparisons. Similar to Fig. 1, we see that in the top of Fig. 8, the optimal
$\beta$ tends to zero as $p$ increases or, equivalently, as the source and
target task become more distinct. At $p=0$, the source and target task
problems coincide yielding an average $\beta$ with low uncertainty. Hence, the
regime where $p$ is sufficiently small, we see that full transfer and optimal
transfer perform similarly while no transfer has significantly large RMSE. As
$p$ increases, the accuracy of full transfer decreases and ultimately becomes
worse than no transfer near $p=2$. Observe that optimal transfer adapts to the
difference between the source and target tasks and subsequently outperforms no
transfer and full transfer over the entire range of $p$. Because of the
sparsity of the target dataset, we see large uncertainty in the RMSE for the
no transfer case as well as the optimal $\beta$ values.

###  3.3 Domain adaptation with PCE models of subsurface scattering

In this practical demonstration, we apply the probabilistic transfer learning
methodology to Bayesian inversion of a 1-dimensional subsurface scattering
model over differing domains. Problems such as this arise in applications such
as geosteering for drilling and resource extraction [47] where determining
properties of strata is paramount. The forward model considers a collection of
$n_{l}$ subsurface layers depicted in the schematic Fig. 9 under the
simplifying assumption that physical properties vary depthwise ($z$) in
layers. The quantities of interest are the electrical resistivities
$R_{i};i=1,\ldots,n_{l}$ for each of the layers, measured in units of ohm-
meter $\mathrm{\Omega}\cdot\mathrm{m}$, as well as the $z$-depth
$z_{j},j=1,\ldots,n_{l}-1$ of the boundary of each layer, measured in
$\mathrm{ft}$. A transmitter and receiver device is placed at the origin which
we take to be $z=0$.

Figure 9: Schematic diagram of the 3-layer subsurface transfer learning
problem with physical model parameters given by the layer resistivities
$R_{1}$, $R_{2}$, $R_{3}$ and boundaries $z_{1},z_{2}$. The target task is
differentiated from the source task by varying the depth $z_{2}$ of the second
boundary or by changing the resistivity of the third layer $R_{3}$. This
represents the geosteering scenario where rock layers have the same
composition by spatially varying thicknesses.

For the sake of interpretability, we consider a $3$-layer subsurface model
with $5$ total parameters consisting of the resistivities $R_{1}$, $R_{2}$,
$R_{3}$ of each layer along with two $z$-depths $z_{1}$, $z_{2}$ that define
the location of the two layer boundaries relative to the sensor. The
generative model is then given by solving Maxwell’s equations over the
1-dimensional domain. We approximate the generative model with a PCE over the
five parameters of maximum degree $d=3$, yielding a total of $56$ coefficients
so that ${\bm{\theta}}\in\mathbb{R}^{56}$. The transfer learning framework
methodology is applied to a parametric sequence of domain adaptation problems
defined by source and target PCE surrogate approximation of the subsurface
model. The source parameters ranges are kept fixed while the target domain is
varied linearly by modifying the thickness of the second layer, which contains
the sensor, or the resistivity range for the third layer. This mimics a
geosteering scenario where drilling has moved from one location to another and
either the subsurface layer thickness or composition is spatially varying.

Similar to Sec. 3.1, this setup allows us to directly measure the performance
gain achieved by transfer learning as a function of the discrepancy between
source and target data distributions. The ranges of the source parameters are
displayed in Table 1. The target domain is shifted in two ways. First, the
range for the boundary $z_{2}$ between layers 2 and 3 is shifted from $[1,2]$
to $[4,5]$ for a maximum difference of $3\hskip 2.84526pt\mathrm{ft}$. Second,
$R_{3}$, the resistivity of the third layer, from $[7,9]$ to $[17,19]$, for a
maximum difference of $10\hskip 2.84526pt\mathrm{\Omega}\cdot\mathrm{m}$. The
source problem is calibrated on $N_{S}=200$ data points uniformly sampled over
the domain in Table 1 while the target is calibrated using only $N_{T}=57$
data points. This sample size is just sufficient to obtain an over-determined
least squares problem for the $56$ PCE parameters. As in Sec. 3.2, we use the
RMSE over the set of validation data to measure the success of transfer
learning.

Table 1: Ranges of model parameters defining the source domain. Parameter | $R_{1}$ | $R_{2}$ | $R_{3}$ | $z_{1}$ | $z_{2}$
---|---|---|---|---|---
Unit | $\mathrm{\Omega}\cdot\mathrm{m}$ | $\mathrm{\Omega}\cdot\mathrm{m}$ | $\mathrm{\Omega}\cdot\mathrm{m}$ | $\mathrm{ft}$ | $\mathrm{ft}$
min | 1 | 4 | 7 | -2 | 1
max | 3 | 6 | 9 | -1 | 2

The results for the shifting $z_{2}$ and $R_{3}$ are displayed in Fig. 10
where we see the validation error and its uncertainty for optimal transfer
along with the two baseline cases of no transfer, and full transfer. As seen
in previous examples, the mean optimal $\beta$ decreases monotonically from
one as the source and target domains become increasingly separated. This shows
that the probabilistic transfer learning methodology is able to diffuse source
knowledge in response to large discrepancies between the two tasks. When the
two domains coincide, we see that no transfer results in a larger RMSE than
both full and optimal transfer of knowledge which continue to behave similarly
as long as the source and target domains remain close. As the domains separate
further, the two tasks become increasingly dissimilar, resulting in
significant increase in the RMSE for full transfer while optimal transfer
begins to adapt and coincide with no transfer. Random sampling results in
significant variance of the target model calibration. Uncertainty in the no
transfer RMSEs is considerably larger due to the sparsity of the target
dataset. Focusing on the mean values, we see that optimal transfer outperforms
no transfer and full transfer over nearly the full sequence of shifted target
domains.

Figure 10: Subsurface inversion: RMSE of target model evaluated with the mean
parameters of the PFP distribution. The cases of no transfer, optimal
transfer, and full transfer are displayed in red, blue, and green,
respectively. Optimal $\beta$ is shown in dashed black. Each quantity is
plotted as a function of the shift between source and target domains. The top
displays the case where $z_{2}$ is shifted, while the bottom shows the
behavior when $R_{3}$ is shifted. The confidence bounds reflect the variance
from random sampling of the source and target domains and are drawn at 1
standard deviation.

A particular aspect of these results that merits further discussion is how the
correlation structure of the source and target task affects the performance of
transfer learning. Fig. 11 displays the matrix of correlation coefficients
across all 56 PCE parameters of the source model after calibration using the
source data. Observe that the matrix displays strong diagonal dominance
suggesting that the source posterior encodes Gaussian distributions for the
parameters with weak correlations. This results from using orthogonal
polynomial basis functions for the PCE expansions which yield uncorrelated
random variables. This lack of correlation structure potentially uncouples and
thus simplifies the transfer learning problem. For example, recall that in
Sec. 2.3, it was noted how power tempering preserves the correlation
structure since it uniformly scales the source likelihood’s covariance matrix.
Hence, while modifying the source likelihood’s overall uncertainty, power
tempering maintains the approximate independence across parameters during the
transfer learning process.

Figure 11: Correlation coefficients of the source likelihood. The color range
was set to span the minimum and maximum values of the off-diagonal entries as
the diagonal entries are 1 by definition. Observe that using an orthonormal
polynomial basis results in likelihood PDFs characterizing approximately
uncorrelated unknown parameters.

##  4 Conclusion

In this paper, we presented a probabilistic transfer learning framework based
on informing a target task using optimal tempering of the Bayesian posterior
of a source task. Transfer learning is invaluable in scenarios with costly
forward evaluations and reservoirs of related data. It is also key in data-
driven modeling problems where sparse and/or noisy data makes model
calibration ill-posed. While transfer learning approaches exist for
deterministic ML models, less focus has been given to uncertainty
quantification. The proposed method fills this need.

We extended transfer learning to Bayesian inference problems through an
optimization-based methodology utilizing objective functions to control the
diffusion of knowledge from source to target task. We demonstrated that the
method transitions between full and no transfer of information while capturing
uncertainty and avoiding negative transfer. The method involved novel
application of several existing information-theoretic and discrepancy metrics
to construct suitable objective functions for optimally tempering the source
task posterior distribution. An extensive investigation of the behavior of
transfer learning under these objective functions was carried out on a series
of domain adaptation problems. This was used to understand factors influencing
the robustness of the methodology and to down-select to the best performing
objective, namely the Expected Data Fit. We showed that a significant
performance gain is achievable using the transfer learning procedure on a task
adaptation problem based on the Ishigami function. Similar performance gains
were seen on a geosteering application based on a 3-layer subsurface
scattering model. The optimal tempering procedure was seen to outperform naive
full transfer, given by a standard Bayesian update, as well as ignoring the
source data.

These results suggest that the proposed methodology can improve calibration
performance in sparse and/or noisy data setting but more investigation is
required to see whether this carries over to large-scale applications such as
those arising in machine learning models. Also investigation of more fine-
grained control of the tempered correlation structure is left for future work.

## Acknowledgements

This work was supported by the U.S. Department of Energy, Office of Science,
SBIR/STTR Programs Office under Award Number DE-SC0021607. It was also
supported by the Laboratory Directed Research and Development program at
Sandia National Laboratories, a multimission laboratory managed and operated
by National Technology and Engineering Solutions of Sandia LLC, a wholly owned
subsidiary of Honeywell International Inc. for the U.S. Department of Energy’s
National Nuclear Security Administration under contract DE-NA0003525. This
report describes objective technical results and analysis. Any subjective
views or opinions that might be expressed in the report do not necessarily
represent the views of the U.S. Department of Energy or the United States
Government.

## References

  * [1] AI Forrester, A. Sobester, and AJ Keane.  Engineering Design via Surrogate Modeling: A Practical Guide.  Wiley, 2008.
  * [2] Udo Von Toussaint.  Bayesian inference in physics.  Reviews of Modern Physics, 83(3):943, 2011.
  * [3] V Dose.  Bayesian inference in physics: case studies.  Reports on Progress in Physics, 66(9):1421, 2003.
  * [4] Nestor V Queipo, Raphael T Haftka, Wei Shyy, Tushar Goel, Rajkumar Vaidyanathan, and P Kevin Tucker.  Surrogate-based analysis and optimization.  Progress in aerospace sciences, 41(1):1–28, 2005.
  * [5] Donald R Jones, Matthias Schonlau, and William J Welch.  Efficient global optimization of expensive black-box functions.  Journal of Global optimization, 13:455–492, 1998.
  * [6] Carl Edward Rasmussen and Christopher KI Williams.  Gaussian processes for machine learning.  MIT press Cambridge, MA, 2006.
  * [7] Timothy W Simpson, Timothy M Mauery, John J Korte, and Farrokh Mistree.  Kriging models for global approximation in simulation-based multidisciplinary design optimization.  AIAA journal, 39(12):2233–2241, 2001.
  * [8] Yongchao Li, Yanyan Wang, and Liang Yan.  Surrogate modeling for bayesian inverse problems based on physics-informed neural networks.  Journal of Computational Physics, 475:111841, 2023.
  * [9] Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis.  Physics-informed neural networks (pinns) for fluid mechanics: A review.  Acta Mechanica Sinica, 37(12):1727–1738, 2021.
  * [10] Norbert Wiener.  The homogeneous chaos.  American Journal of Mathematics, 60(4):897–936, 1938.
  * [11] Dongbin Xiu and George Em Karniadakis.  The wiener–askey polynomial chaos for stochastic differential equations.  SIAM journal on scientific computing, 24(2):619–644, 2002.
  * [12] RG Ghanem and PD Spanos.  Stochastic finite elements: A spectral approach (springer-verlag, new york, 1991).
  * [13] Oliver G Ernst, Antje Mugler, Hans-Jörg Starkloff, and Elisabeth Ullmann.  On the convergence of generalized polynomial chaos expansions.  ESAIM: Mathematical Modelling and Numerical Analysis, 46(2):317–339, 2012.
  * [14] Jun S Liu and Jun S Liu.  Monte Carlo strategies in scientific computing, volume 75.  Springer, 2001.
  * [15] Youssef M Marzouk, Habib N Najm, and Larry A Rahn.  Stochastic spectral methods for efficient bayesian solution of inverse problems.  Journal of Computational Physics, 224(2):560–586, 2007.
  * [16] Géraud Blatman and Bruno Sudret.  Adaptive sparse polynomial chaos expansion based on least angle regression.  Journal of computational Physics, 230(6):2345–2367, 2011.
  * [17] Dongbin Xiu and George Em Karniadakis.  Modeling uncertainty in flow simulations via generalized polynomial chaos.  Journal of computational physics, 187(1):137–167, 2003.
  * [18] Habib N Najm.  Uncertainty quantification and polynomial chaos techniques in computational fluid dynamics.  Annual review of fluid mechanics, 41:35–52, 2009.
  * [19] Xujia Zhu and Bruno Sudret.  Stochastic polynomial chaos expansions to emulate stochastic simulators.  International Journal for Uncertainty Quantification, 13(2), 2023\.
  * [20] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009.
  * [21] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  Proceedings of the IEEE, 109(1):43–76, 2020.
  * [22] Mei Wang and Weihong Deng.  Deep visual domain adaptation: A survey.  Neurocomputing, 312:135–153, 2018.
  * [23] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu.  A survey on deep transfer learning.  In Artificial Neural Networks and Machine Learning–ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27, pages 270–279. Springer, 2018.
  * [24] Hal Daumé III.  Frustratingly easy domain adaptation.  arXiv preprint arXiv:0907.1815, 2009.
  * [25] Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang.  Domain adaptation via transfer component analysis.  IEEE transactions on neural networks, 22(2):199–210, 2010.
  * [26] Wenyuan Dai, Gui-Rong Xue, Qiang Yang, and Yong Yu.  Co-clustering based classification for out-of-domain documents.  In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 210–219, 2007.
  * [27] Neil D Lawrence and John C Platt.  Learning to learn with the informative vector machine.  In Proceedings of the twenty-first international conference on Machine learning, page 65, 2004.
  * [28] Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He.  Supervised representation learning: Transfer learning with deep autoencoders.  In Twenty-fourth international joint conference on artificial intelligence, 2015.
  * [29] Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He.  Supervised representation learning with double encoding-layer autoencoder for transfer learning.  ACM Transactions on Intelligent Systems and Technology (TIST), 9(2):1–17, 2017.
  * [30] Roger G Ghanem and Pol D Spanos.  Stochastic finite elements: a spectral approach.  Springer, 1991.
  * [31] Christian Soize and Roger Ghanem.  Physical systems with random uncertainties: chaos representations with arbitrary probability measure.  SIAM Journal on Scientific Computing, 26(2):395–410, 2004.
  * [32] Thierry Crestaux, Olivier Le Maıtre, and Jean-Marc Martinez.  Polynomial chaos expansion for sensitivity analysis.  Reliability Engineering & System Safety, 94(7):1161–1172, 2009\.
  * [33] Mohammad Hadigol and Alireza Doostan.  Least squares polynomial chaos expansion: A review of sampling strategies.  Computer Methods in Applied Mechanics and Engineering, 332:382–407, 2018.
  * [34] Joseph G Ibrahim, Ming-Hui Chen, and Debajyoti Sinha.  On optimality properties of the power prior.  Journal of the American Statistical Association, 98(461):204–213, 2003.
  * [35] Joseph G Ibrahim, Ming-Hui Chen, Yeongjin Gwon, and Fang Chen.  The power prior: theory and applications.  Statistics in medicine, 34(28):3724–3749, 2015.
  * [36] Christopher M Bishop and Nasser M Nasrabadi.  Pattern recognition and machine learning.  Springer, 2006.
  * [37] Solomon Kullback and Richard A Leibler.  On information and sufficiency.  The annals of mathematical statistics, 22(1):79–86, 1951.
  * [38] David JC MacKay.  Information theory, inference and learning algorithms.  Cambridge university press, 2003.
  * [39] Alex P Zijdenbos, Benoit M Dawant, Richard A Margolin, and Andrew C Palmer.  Morphometric analysis of white matter lesions in mr images: method and validation.  IEEE transactions on medical imaging, 13(4):716–724, 1994.
  * [40] Ruey-Pyng Lu, Eric P Smith, and IJ Good.  Multivariate measures of similarity and niche overlap.  Theoretical Population Biology, 35(1):1–21, 1989.
  * [41] T. Butler, J. Jakeman, and T. Wildey.  Combining push-forward measures and bayes’ rule to construct consistent solutions to stochastic inverse problems.  SIAM Journal on Scientific Computing, 40(2):A984–A1011, 2018.
  * [42] Vladimir Igorevich Bogachev and Maria Aparecida Soares Ruas.  Measure theory, volume 1.  Springer, 2007.
  * [43] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  The Journal of Machine Learning Research, 22(1):2617–2680, 2021\.
  * [44] Kaare Brandt Petersen, Michael Syskind Pedersen, et al.  The matrix cookbook.  Technical University of Denmark, 7(15):510, 2008.
  * [45] Tsutomu Ishigami and Toshimitsu Homma.  An importance quantification technique in uncertainty analysis for computer models.  In [1990] Proceedings. First international symposium on uncertainty modeling and analysis, pages 398–403. IEEE, 1990.
  * [46] Ilya M Sobol and Yu L Levitan.  On the use of variance reducing multipliers in monte carlo computations of a global sensitivity index.  Computer Physics Communications, 117(1):52–61, 1999.
  * [47] WG Lesso Jr, SV Kashikar, et al.  The principles and procedures of geosteering.  In SPE/IADC Drilling Conference. Society of Petroleum Engineers, 1996.