# Out-of-Variable Generalization

Siyuan Guo
&Jonas Wildberger 22footnotemark: 2
&Bernhard Schölkopf 22footnotemark: 2
Correspondence to<EMAIL_ADDRESS>Planck Institute for Intelligent
Systems, Tübingen, GermanyDepartment of Computer Science, University of
Cambridge, United Kingdom

###### Abstract

The ability of an agent to perform well in new and unseen environments is a
crucial aspect of intelligence. In machine learning, this ability is referred
to as strong or out-of-distribution generalization. However, simply
considering differences in data distributions is not sufficient to fully
capture differences in environments. In the present paper, we assay out-of-
variable generalization, which refers to an agent’s ability to handle new
situations that involve variables never jointly observed before. We expect
that such ability is important also for AI-driven scientific discovery:
humans, too, explore "Nature" by probing, observing and measuring subsets of
variables at one time. Mathematically, it requires efficient re-use of past
marginal knowledge, i.e., knowledge over subsets of variables. We study this
problem, focusing on prediction tasks that involve observing overlapping, yet
distinct, sets of causal parents. We show that the residual distribution of
one environment encodes the partial derivative of the true generating function
with respect to the unobserved causal parent. Hence, learning from the
residual allows zero-shot prediction even when we never observe the outcome
variable in the other environment.

##  1 Introduction

An animal’s natural intelligence differs from machine intelligence in their
ability to generalize from one problem to the next. Much of modern machine
learning can be seen as large scale pattern recognition on suitably collected
independent and identically distributed (i. i. d.) data. Its success boils
down to its ability to generalize from a set of data points to the next one
sampled from the same distribution. Though both using the term generalization,
generalization in the i. i. d. setting is not the same as generalization from
one problem to the next. The machine learning community studies the latter
under the term out-of-distribution generalization [52, 24, 48, 2, 31, 66, 64],
where training and test data deviate in their distributions. However,
differences in distributions may not fully describe differences in
environments [6]. In the present work, we investigate generalization across
environments containing different sets of variables, referring to the problem
as out-of-variable (OOV) generalization.

Out-of-variable generalization aims to transfer knowledge learned from a set
of source environments to a target environment that contains variables never
jointly present in any of the source environments. This type of learning
requires synthesizing information from multiple sources that contain different
sets of variables. For example, human doctors can often accurately diagnose a
patient with a disease even if their symptoms, medical history and set of
available clinical tests differ from past patients, whereas machine learning
models struggle in prediction when facing overlapping, yet distinct, sets of
variables. We can think of ’Nature’ as a complex system of interrelated
processes and variables. Humans, due to limited resources and knowledge, only
observe and measure a subset of variables at a time. Scientific discovery [50,
23] can thus be seen as a problem in which scientist need to generalize out-
of-variable. It is crucial to understand how and to what extent knowledge can
be re-utilized.

Recent work in transfer learning [59, 63, 34, 18, 65] has focused on the
development of discriminative models. Such models are preferred [57] over
generative models in prediction tasks as they more directly solve the task at
hand. They learn a direct map from inputs $X$ to the target variable $Y$,
aiming to find $\mathbb{E}[Y\mid X]$. Common strategies are to either extract
re-usable features [33, 38, 56] or fine-tune model parameters [15, 51, 25, 12,
60] across tasks. Empirical performance shows such reuse of knowledge
generally performs better than starting from a random state and is also more
data-efficient. Marginal problems [58, 53, 44, 27, 36, 17, 20], on the other
hand, often apply generative modelling approaches, i.e., to search joint
distributions that are compatible with marginal observations. Out-of-variable
generalization combines marginal problems and transfer learning. The goal is
to re-use marginal observations for prediction tasks. In the present work, we
propose a method that generalizes to variables never jointly present in the
source environments more efficiently than by modelling the joint distribution.
Our contributions are:

  * •

We introduce and formalize out-of-variable generalization (§ 2) and
contextualize this concept within the existing literature (§ 5).

  * •

We prove impossibility and possibility theorems to show limitations of
discriminative approaches in out-of-variable scenarios (§ 3.2).

  * •

Modelling moments from the residual distribution of the source environment has
predictive power for variables never jointly observed before. Via a motivating
example (§ 3.3), we propose a method (§ 3.4) for general nonlinear smooth
functions. We find in experiments (§ 4) that our approach achieves strong
zero-shot transfer performance.

Fig. 5 provides a toy examples giving an overview of the main idea.

$X_{1}$$X_{2}$$X_{3}$$Y$ (a) From Marginal to Marginal

Figure 1: Proposed

Figure 2: Optimal

Figure 3: Marginal

Figure 4: FineTune

Figure 5: Toy example: (a) the blue box includes variables observed in the
source environment, and the orange box those in the target environment. A
directed edge represents a causal relationship between two variables. The goal
is to improve the zero-shot (i.e., without additional data) prediction of $Y$
in the target environment using the source environment. (b)-(e) Learning from
the residual distribution in the source environment, our proposed method
exhibits zero-shot performance close to the optimal solution compared to two
baselines, as shown via the predictors’ contour plots (for details, cf. §
4.1).

##  2 Out-of-Variable Generalization

###  2.1 Motivation

######  Definition 1 (Structural Causal Model (SCM))

A structural causal model $\mathcal{M}$ consists of a collection of random
variables $X_{1},\dots,X_{n}$ and its corresponding structural assignments

$X_{i}:=f_{i}(\textbf{PA}_{i},U_{i}),\forall i=1,\ldots,n,$ (1)

where $\textbf{PA}_{i}$ is the parents or direct causes of $X_{i}$ and $U_{i}$
are jointly independent noise variables. Given a SCM $\mathcal{M}$, one can
find its corresponding directed acyclic graph (DAG) where each node has all
its incoming edges from its parents set. Any joint distribution generated from
some SCM $\mathcal{M}$ satisfying the DAG structure $\mathcal{G}$ allows the
Markov factorization

$p(x_{1},\ldots,x_{n})=\prod_{i=1}^{n}p(x_{i}\mid\textbf{pa}_{i}^{\mathcal{G}})$
(2)

where $\textbf{pa}_{i}^{\mathcal{G}}$ are parents of node $x_{i}$ in graph
$\mathcal{G}$.

######  Principle 1 (Independent Causal Mechanisms (ICM))

A change in one mechanism $p(x_{i}\mid\textbf{pa}_{i}^{\mathcal{G}})$ does not
inform [21, 27] or influence [47] any of the other mechanisms
$p(x_{j}\mid\textbf{pa}_{j}^{\mathcal{G}})(i\neq j)$.

Structural causal model (SCM) [39] establishes a mathematical framework
modelling causal relationships exist in Nature and ICM principle [49]
postulates that true causal mechanisms should be independent of each other.
Prior work [1, 22, 4, 45] about causality and generalization exploit the
invariance property of causal mechanisms across domains. Beyond causal
frameworks, learning distributed representations [8, 55], e.g. taking the form
of Eq. 2, has also long been argued to offer advantages in efficient parameter
estimation [30, 11] and interpretability [28, 32]. In this paper, we highlight
another advantage:

The Markov factorization frees us from the need of observing all variables of
interest at the same time.

Observation For each observed variable $i$, suppose there exists some
environment that contains $X_{i}$ and its causal parents
$\textbf{PA}_{i}^{\mathcal{G}}$. Then by Eq. 2, one can recover the joint
distribution by piecing together learnt conditional distributions
$p(x_{i}\mid\textbf{pa}_{i}^{\mathcal{G}})$ in each environment, even though
there may never exist an environment that contains all variables of interest
at once. This phenomenon also occurs in undirected probabilistic graphical
models [30], where the joint distribution111The joint distribution can be
represented as
$p(x_{1},\dots,x_{n})=\frac{1}{Z}\prod_{c\in\text{cliques}}\Psi_{c}(x_{c})$.
is recoverable given potentials learnt from environments that only contain
variables appearing in each clique.

###  2.2 OOV Generalization

Formally, an environment $\mathcal{E}$ consists of a domain $\mathcal{D}$ and
a task $\mathcal{T}$. The domain is defined by two components: a feature
variable space, denoted as $\mathcal{X}:=(X_{1},X_{2},\dots)$, and its
corresponding marginal probability distribution, represented by
$P(\mathcal{X})$. Given a domain, a task is defined by: a target variable
space, represented by $\mathcal{Y}:=Y$, and a predictive function, represented
as $f(\cdot)$. The variable space in an environment contains both the feature
and target variable spaces. Note, in our setting, there could be multiple
source environments, e.g. $\mathcal{E}_{S}:=\\{\mathcal{E}_{s}\\}_{s\in S}$,
whereas the target environment only comprises a single environment
$\mathcal{E}_{T}$. To differentiate between components belonging to the source
and target environments, the subscripts $s$ and $T$ are used, respectively.

What does it mean to have same variables? In probability and statistics, one
considers a random variable as a function that maps the outcome of an
experiment to a real number. It is equipped with a probability space,
$(\Omega,\mathcal{F},\mathbb{P})$. One can interpret the space as a model for
experiment where $\Omega$ is the set of possible outcomes, $\mathcal{F}$ is
the set of observable sets of outcomes, and $\mathbb{P}(A)$ is the probability
of an event $A$. In practice, random variables may share the same physical
meaning but have different distributions, e.g. blood pressure measured in
different age groups. Under certain pathological case, one can even construct
random variables that have different physical meanings but share the same
mathematical object. For example, the risk of a disease is governed by law in
Nature relating to clinical variables, like blood pressure, heart rate and
lifestyle. Such law remains the same across different patients in different
hospitals. The variables recorded in each hospital share the same physical
meanings, though may differ in distributions. Such inconsistency between
statistical and physical interpretation of variables motivates us to formalize
what we mean by out-of-variables.

###### Definition 2

Two random variables share physical meaning if they represent the same
physical property [9] of the same physical system [7]. Further, they also
share the same measurement process. We say $\mathcal{A}$ is not a subset of
$\mathcal{B}$, if there exists a random variable $X$ in $\mathcal{A}$ such
that it does not share physical meaning with any of the random variables in
$\mathcal{B}$.

######  Definition 3 (OOV Generalization)

Out-of-variable uncertainty arises when the variable space of source
environment(s) differs from the variable space of the target environment, i.e
$\forall s\in
S,\\{\mathcal{X}_{T},\mathcal{Y}_{T}\\}\not\subseteq\\{\mathcal{X}_{s},\mathcal{Y}_{s}\\}$.
The need for generalization happens when we want to improve a quantity $Q_{T}$
in the target environment using the related information from source
environment(s).

For this paper, $Q_{T}=f_{T}(\cdot)$, the target predictive function. Appendix
A provides a detailed discussion of the need to introduce Def. 2 and its
connection with statistics and physics. Though the applicability of the work
set out in this paper is not limited to a causal framework, for ease of
illustration, from here onwards we consider variables are causally related
with each other.

##  3 Transfer Learning From Residual

###  3.1 Problem Formulation

Consider a structural causal model that is generated by an additive noise
model (ANM):

$\displaystyle\begin{split}&Y:=\phi(X_{1},X_{2},X_{3})+\epsilon\\\
&X_{1}:=U_{1},\ X_{2}:=U_{2},\ X_{3}:=U_{3}\end{split}$ (3)

where $\phi$ is some unknown function and $U_{1},U_{2},U_{3}$ are jointly
independent random variables and $\epsilon\sim\mathcal{N}(0,\sigma^{2})$.
Assume we do not have access to the joint environment $J$ that contains all
variables of interest, namely $(X_{1},X_{2},X_{3},Y)$. Instead, we consider:

  * •

Source environment $S$ with variables $(X_{1},X_{2},Y)$

  * •

Target environment $T$ with variables $(X_{2},X_{3},Y)$

The optimal predictive functions achievable via minimizing mean squared error
loss are $f_{S}(x_{1},x_{2})=\mathbb{E}[Y\mid x_{1},x_{2}]$ and
$f_{T}(x_{2},x_{3})=\mathbb{E}[Y\mid x_{2},x_{3}]$, where each corresponds to
a mapping learnt under discriminative modelling from environment $S$ and $T$
respectively. We consider the residual distribution from the source
environment as the distribution of $Y-f_{S}(X_{1},X_{2})\mid X_{1},X_{2}$. The
optimal predictive function satisfies marginal consistency condition:

$\displaystyle\mathbb{E}_{X_{1}}[f_{S}(x_{1},x_{2})]=\mathbb{E}_{X_{3}}[f_{T}(x_{2},x_{3})]$
(4)

Suppose $Y$ from the target environment is never observed. The challenge is
that directly training a classifier in the target environment is not possible.
Our goal is to learn the target predictive function $f_{T}$ in a data-
efficient way by transferring knowledge learned from the source environment.

###  3.2 Theorem: Possibility and Impossibility

In this section, we show limitations of discriminative modelling, which
considers $\mathbb{E}[Y\mid X]$ only without taking into account of its
residual distribution. Under such modelling, Theorem 3.1 shows it is
impossible to identify the optimal predictive function in the target
environment for scenarios shown in Fig. 6. In this context, non-
identifiability means non-uniqueness of a target function satisfying marginal
consistency conditions (as described in Eq. 4). See Appendix C.1 and C.2 for
detailed proofs.

######  Theorem 3.1 (Impossibility Theorem)

Consider out-of-variable scenarios illustrated in Fig. 6 (a)-(c). They are
governed by the structural causal model described in § 3.1. Suppose all
variables considered are continuous variables, except $X_{1},X_{3}$ in Fig.
6(c) are binary. Assume 1) for all $i$, $p(x_{i})$ is known and let the
support set be $S_{i}:=\\{x\in\mathbb{R}\mid p_{i}(B(x,\epsilon))>0\\}$ where
$p_{i}$ denotes the marginal distribution of $x_{i}$ and $B(x,\epsilon)$ is
the ball centred around $x$ with radius $\epsilon$. 2) Suppose for all $i$
there exists two distinct points $x,x^{\prime}\in S_{i}$. Given fixed $f_{S}$,
for any function $f_{T}$ that satisfies
$\mathbb{E}_{X_{1}}[f_{S}(x_{1},x_{2})]=\mathbb{E}_{X_{3}}[f_{T}(x_{2},x_{3})]$,
there always exists a different function $f_{T}^{\prime}$ such that it
satisfies the above condition and for any chosen $R>0$,
$\|f_{T}-f_{T}^{\prime}\|_{2}\geq R$.

Theorem 3.1 says enforcing marginal consistency between source and target
predictive function is (in general) not sufficient to identify a unique
solution, and the learnt solution may deviate arbitrarily large from the
optimal one. While such impossibility theorem is discouraging, Theorem 3.2
illustrates special scenarios where exploiting information contained in
$\mathbb{E}[Y\mid X]$ for each environment is sufficient to identify the
optimal predictive function in the target environment.

###### Theorem 3.2

Consider a target variable $Y$ and its corresponding causal parents
$\textbf{PA}_{Y}$. Let $Z$ be a common child of all variables in
$\textbf{PA}_{Y}$ and it satisfies $Z\perp\\!\\!\\!\perp
Y\mid\textbf{PA}_{Y}$. Given:

  * •

Environment $S_{1}$ contains variables $(Z,Y)$ and
$f_{S_{1}}(z)=\mathbb{E}[Y\mid Z]$,

  * •

Environment $S_{2}$ contains variables $(\textbf{PA}_{Y},Z)$ and
$f_{S_{2}}(\textbf{pa}_{Y})=\mathbb{E}[Y\mid\textbf{PA}_{Y}]$

Assume the SCM is generated by an additive noise model and $f_{S_{2}}$ is an
invertible function. With small noise in the functional relationship between
$Z$ and $\textbf{PA}_{Y}$, composing $f_{S_{1}}\cdot f_{S_{2}}$ identifies the
optimal predictive function in the target environment that contains variables
never jointly observed previously, i.e., $(\textbf{PA}_{Y},Y)$. Fig. 7 in
Appendix C.2 shows an example of such scenario where $Z=X_{1}$ and
$\textbf{PA}_{Y}=X_{2}$.

$X_{1}$$X_{2}$$X_{3}$$Y$

(a) Marginal to Marginal

$X_{1}$$X_{2}$$X_{3}$$Y$

(b) Marginal to Joint

$X_{1}$$X_{2}$$X_{3}$$Y$

(c) Merge Datasets

Figure 6: Examples of OOV Scenarios

Näive Model Given the problem described in § 3.1, one approach is to train
separate models for each variable that is observed, e.g. $(X_{1},Y)$ and
$(X_{2},Y)$. This hopes to additively re-use the model trained on the
overlapping variable $X_{2}$. When facing the prediction task with variables
$(X_{2},X_{3},Y)$, the remaining step is to train a model on ($X_{3},Y)$. One,
then, hopes to recover the optimal predictive function by flexibly composing
trained models on different sets of variables. This approach also circumvents
the need to jointly observe variables of interest $(X_{2},X_{3},Y)$. Here we
highlight, this method relies on the fact that there is no interaction between
different variables. We show the model in detail in Appendix B. Above argument
still applies when variable considered are replaced to any subsets of
variables.

In this paper, we ask: how and to what extent can we generalize given we do
not observe all variables of interest. We illustrate our method via a
motivating example § 3.3 with details in § 3.4. The idea is: 1) general to
work for scenarios in Fig.6, and 2) is not restricted to strict causal
assumptions, e.g. covariates need not be strict causal parents.

###  3.3 Motivating example

Consider the problem described in § 3.1 where $\phi$ is a polynomial function:

$Y:=\alpha_{1}X_{1}+\alpha_{2}X_{2}+\alpha_{3}X_{3}+\alpha_{4}X_{1}X_{2}+\alpha_{5}X_{1}X_{3}+\alpha_{6}X_{2}X_{3}+\alpha_{7}X_{1}X_{2}X_{3}+\epsilon$
(5)

Let $X_{i}$ has mean $\mu_{i}$, variance $\sigma_{i}$ and skew $k_{i}$ for all
$i$. The optimal predictive functions for environment $S$ and $T$ are:

$\displaystyle\begin{split}f_{S}(x_{1},x_{2})&=(\alpha_{3}\mu_{3})+(\alpha_{1}+\alpha_{5}\mu_{3})x_{1}+(\alpha_{2}+\alpha_{6}\mu_{3})x_{2}+(\alpha_{4}+\alpha_{7}\mu_{3})x_{1}x_{2}\\\
f_{T}(x_{2},x_{3})&=(\alpha_{1}\mu_{1})+(\alpha_{3}+\alpha_{5}\mu_{1})x_{3}+(\alpha_{2}+\alpha_{4}\mu_{1})x_{2}+(\alpha_{6}+\alpha_{7}\mu_{1})x_{2}x_{3}\end{split}$
(6)

A common strategy in transfer learning is to fine-tune model parameters. We
see from the above example that such strategy is reasonable, as the
coefficients in each term does overlap in Eq. 6. We can thus hope that the
model parameters can be adapted quickly when training in the target
environment. Though it is clear that we cannot uniquely determine the
coefficients of $f_{T}$ without observing any data from environment $T$. Since
even with known representation, the system of equations is under-determined
with eight unknown coefficients and four estimated values.

To illustrate our idea, consider the skew from the residual distribution in
environment $S$:

$\displaystyle\mathbb{E}\big{[}(Y-f_{S}(x_{1},x_{2})^{3}\mid
x_{1},x_{2}\big{]}$
$\displaystyle=(\alpha_{3}+\alpha_{5}x_{1}+\alpha_{6}x_{2}+\alpha_{7}x_{1}x_{2})^{3}\mathbb{E}[(X_{3}-\mu_{3})^{3}]$
(7)

We observe the term in bracket coincides exactly with the first-order partial
derivative of the generating function $\phi$ with respect to variable $X_{3}$
evaluated at $x_{1},x_{2},\mu_{3}$:

$\displaystyle\frac{\partial\phi}{\partial
X_{3}}\bigg{|}_{x_{1},x_{2},\mu_{3}}$
$\displaystyle=\alpha_{3}+\alpha_{5}x_{1}+\alpha_{6}x_{2}+\alpha_{7}x_{1}x_{2}$
(8)

Under the polynomial above, let $Z=\frac{1}{k_{3}}(Y-f_{S}(x_{1},x_{2}))^{3}$
and $g(x_{1},x_{2})=[x_{1},x_{2},x_{1}x_{2}]$. Learning $\theta$ from fitting
$Z$ with $(\theta^{T}g(x_{1},x_{2}))^{3}$ enables estimation of coefficients
$\alpha_{3},\alpha_{5},\alpha_{6},\alpha_{7}$. Combining with the coefficients
estimated from learning $f_{S}$, we are able to recover the exact coefficient
about $f_{T}$ without the need to train data from environment $T$.

Discussion The intuition behind this seemingly surprising ability is rather
straightforward. The variable $X_{3}$ though unobserved by the source
environment is a generating factor of the target variable $Y$. Its information
is not only contained in the marginalized mean but also in the residual
distribution. Looking at moments of the residual distribution provides an
additional set of equation that helps to determine exact coefficients. Even
without the polynomial assumption, the moment also provides additional
information about the partial derivative of the generating function with
respect to $X_{3}$. This is learnable as the skew of our current residual
distribution is a propagated effect of the skew of $X_{3}$ with some
interference by the noise variable. We can estimate the skew of $X_{3}$ from
samples in the target environment without the need to jointly observe its
corresponding $Y$. Such phenomenon, in fact, extends to general nonlinear
smooth functions.

###  3.4 Method

###### Theorem 3.1

Consider the problem setup in § 3.1. Assume the generating function $\phi$ is
2-times differentiable with respect to the third covariate everywhere and
source environment $S$ contains variables $(X_{1},X_{2},Y)$ with
$f_{S}(x_{1},x_{2})=\mathbb{E}[Y\mid x_{1},x_{2}]$. For fixed $x_{1},x_{2}$,
approximate the function $\phi:x_{1}\times
x_{2}\times\mathcal{X}_{3}\rightarrow\mathbb{R}$ with first-order Taylor
approximation, we have:

$\mathbb{E}[(Y-f_{S}(x_{1},x_{2}))^{n}\mid
x_{1},x_{2}]=\sum_{k=0}^{n}{n\choose
k}\mathbb{E}[\epsilon^{k}](\frac{\partial\phi}{\partial
X_{3}}\bigg{|}_{x_{1},x_{2},\mu_{3}})^{n-k}\mathbb{E}[(X_{3}-\mu_{3})^{n-k}]$
(9)

When $n=3$:

$\mathbb{E}[(Y-f_{S}(x_{1},x_{2}))^{3}\mid
x_{1},x_{2}]=(\frac{\partial\phi}{\partial
X_{3}}\bigg{|}_{x_{1},x_{2},\mu_{3}})^{3}\mathbb{E}[(X_{3}-\mu_{3})^{3}]+\mathbb{E}[\epsilon^{3}]$
(10)

Intuition Under first-order Taylor approximation about the underlying
generating function, the moments of the residual distribution is a mixed
effect between the moments of noise variable and the propagated effects caused
by the unobserved variable. In particular, when $n=3$, most terms involving
undesired noise variable disappear which allows us to estimate desired
information. See Appendix C.3 for detailed proof.

Discussion One can consider this first-order approximation to the true
generating function as a product of the function learned in the source
environment with a linear interaction with the unobserved variable. This means
if the true generating function is indeed linear with respect to the
unobserved variable, for example when $\phi$ is as in § 3.3, then our solution
will be exact. This results in a function space that respects any general
function with respect to the observed variables $X_{1},X_{2}$ while varying
linearly in $X_{3}$, i.e., $H(x_{1},x_{2})\times L(x_{3})$ where $L$ is a
linear function of $X_{3}$. Under model misspecification [61], our method
finds a function in this function space that is closest to the true generating
function. For ease of illustration, we demonstrate under first-order Taylor
approximation with a single unobserved variable. This approach provides us the
flexibility to choose the trade-off between approximation accuracy and
computational efficiency. Model users can determine the degree of
approximation accuracy by choosing the number of moments to model, as well as
the degree of the Taylor approximations. See Appendix D.1 and D.2 for
extensions of our method to higher-order approximations and to situations that
involve multiple unobserved variables.

####  3.4.1 Zero-shot Learning

Applying Theorem 3.1, we propose zero-shot estimate $\tilde{f}_{T}$ of optimal
predictive function in the target environment:

$\tilde{f}_{T}(x_{2},x_{3})=\frac{1}{n}\sum_{i=1}^{n}f_{S}(x_{1,i},x_{2})+h_{\theta}(x_{1,i},x_{2})(x_{3}-\mu_{3}),\quad\text{where}\
x_{1,i}\sim p(x_{1})$ (11)

where $h_{\theta}$ is a neural network parameterized by $\theta$ and is learnt
to estimate the partial derivative via modelling the third moment of the
residual distribution from the source environment. One can consider the
proposed method as a first-order approximation to the true generating function
around its third variable. This will thus be strictly better than näively
marginalizing $f_{S}$ from the source environment. Further the proposed
estimate automatically satisfies the marginal consistency condition as the
second term in Eq. 11 vanishes by taking expectation, i.e.,
$\mathbb{E}_{X_{3}}[\tilde{f}_{T}(x_{2},x_{3})]=\mathbb{E}_{X_{1}}[f_{S}(x_{1},x_{2})]$.
Interestingly, our result shows even though identification may be impossible,
there is prediction power from marginal observations. See Algorithm 1 for
detailed procedure and Appendix E for derivation.

Input : Source environment $S$ with variables $X_{1}$, $X_{2}$ and $Y$; Target
environment $T$ with variables $X_{2}$ and $X_{3}$.

Output : Zero-shot predictive function $\tilde{f}_{T}(x_{2},x_{3})$

1 Step 1: Learn $\mathbb{E}[Y\mid X_{1},X_{2}]$

2 Train a neural network $f_{S}$ via minimizing its mean squared error
$||Y-f_{S}(x_{1},x_{2})||_{2}^{2}$

3 Step 2: Learn partial derivative $h_{\theta}$ from modelling conditional
skew

4 Compute $Z=(Y-f_{S}(X_{1},X_{2}))^{3}$.

5 Estimate the skew of $X_{3}$: $k_{3}=\mathbb{E}[(X_{3}-\mu_{3})^{3}]$, where
$\mu_{3}=\mathbb{E}[X_{3}]$.

6 Train a neural network $h_{\theta}$ via minimizing
$||Z-k_{3}h_{\theta}(x_{1},x_{2})^{3}||_{2}^{2}$

7 Step 3: Monte Carlo Estimation

8 Uniformly sample $n$ observations of $X_{1}$ from Env $S$:
$\\{x_{1,i}\\}_{i=1}^{n}$.

For fixed $x_{2},x_{3}$, calculate the proposed zero-shot estimate in Eq. 11.

Algorithm 1 Zero-shot learning

##  4 Experiments

We perform experiments to evaluate our algorithm’s zero-shot learning
performance. We find that our method demonstrates both qualitative and
quantitative empirical success. We compare the proposed method with several
baselines:

  * •

Optimal: Model learnt if there is sufficient data from the target environment

  * •

Marginal: Marginalize model learnt from the source environment to overlapping
variable(s)

  * •

FineTune: Model learnt from the source environment (zero-shot prediction at
the start of the fine-tuning process)

###  4.1 Synthetic datasets

We generate synthetic data described in § 3.1 where for all $i$,
$X_{i}\sim\Gamma(1,1)$ and $\epsilon\sim\mathcal{N}(0,\sigma^{2})$ with
$\sigma=0.1$. We standardize the covariates into range $[0,5]$ and generate
$100k$ observations for the source environment and $50$ covariates samples
from the target environment, i.e., there is no observation of $Y$ in the
target environment. To learn $f_{S}$ and the partial derivative $h_{\theta}$,
we use a 2-layer MLP with ReLU activation function. All MLPs are trained to
minimize its mean squared error loss between the target and predicted values.
We sample $n=1000$ observations of $X_{1}$ from the source environment to
perform monte-carlo approximation according to Eq. 11.

Qualitative Fig. 5 (b)-(e) shows qualitative comparisons for contour plots of
the zero-shot prediction of our proposed method compared to optimal and the
other two baselines. We see our method is almost identical to the optimal
baseline, whereas the other two baselines deviate far from optimal. This means
our method achieves similar prediction performance to that trained on the
target environment with sufficient data. This shows learning the third moment
of the residual distribution in the source environment is sufficient under
certain function classes to achieve accurate zero-shot estimation.

Quantitative To evaluate our method’s performance on general nonlinear
functions, we generate functions in each function class over $5$ random seeds.
We then aggregate results uniformly over polynomial, nonlinear and
trigonometric functions (described in Appendix F). We evaluate the
quantitative sample efficiency advantage brought by zero-shot estimation of
our method. In the figure from the right-hand side, we compare the number of
samples that contain joint variables required to observe from the target
environment (x-axis) with the percentage advantage offered by zero-shot
estimation instead of training from scratch. Loss (y-axis) is calculated as
the percentage difference between zero-shot loss estimated by a specific
method $(\text{loss}_{m})$ and loss achieved over training on the number of
samples observed $(\text{loss}_{o})$, i.e.,
$(\text{loss}_{m}-\text{loss}_{o})/\text{loss}_{o}$. This means the lower the
loss, the larger the advantage of our proposed method over a new model trained
from scratch in the target environment on the given number of samples. We can
see our method far outperforms the baselines while maintaing tighter error
bars. This shows the generality of our proposed method on different function
classes and the stability of our estimations. In particular, the method offers
significant benefit over training from scratch until a sufficient number of
data points have been collected in the target environment.

##  5 Related Work

Missing data [46] studies the problem when covariates are missing for
individual data points. Most approaches [16, 14, 29] either omit data points
that contain missing values or aim to impute with estimated values. Our
problem differs in that we consider variables that are missing entirely from
certain environments, but we still aim to do zero-shot prediction – which
neither of these two approaches covers.

Marginal problems in the classical statistical [58] and causal literature [26,
36, 20, 17, 44] study the problem of learning over marginal models to infer
properties about the joint model. In contrast with common approaches that
enforce compatibility over joint distributions [36, 26], our work shows
learning from the residual distribution is sufficient to achieve strong
transfer performance and specifies the connection between residual
distributions and functions.

Generalization has predominantly focused on out-of-distribution generalization
[3, 31, 66, 2]. Causality-related approaches either study the transfer of
causal effects across different experimental conditions [40, 6, 41, 5, 13] or
aim to learn invariant causal information that are robust in different
environments [4, 35, 22, 42, 43, 45] . Our work differs from the former as we
study generalization problem to variables never jointly observed before and
the latter implicitly assumes the causal information shared fully characterize
the task of prediction. In our problem, however, different environments share
fundamentally partial but related information. Thus, learning only shared
information as shown by our marginal baseline is inefficient to achieve good
transfer performance.

##  6 Discussion

Limitations and Extensions The performance criteria in Def. 3 is defined as
the target predictive function. Here we highlight it can be extended to other
criteria such as causal structure learning as investigated by [36, 62]. We
also think it would be interesting to extend the notion of variables beyond
observable and measurable quantities (e.g. latent variables). Our method is
now demonstrated under first-order approximations with one unobserved
variable. For extensions to achieve higher approximation accuracy and more
unobserved variables, see Appendix D.

Conclusion In this paper, we introduced out-of-variable generalization. We
show that learning moments from the residual distribution in the source
environment allows us to achieve strong zero-shot performance on variables
never jointly observed before and is efficient to estimate given complex
datasets. We hope our work inspires future work to explore and develop
methodologies that could apply to different out-of-variable scenarios.

## References

  * Pet [2016] Causal inference using invariant prediction: identification and confidence intervals.  _Journal of the Royal Statistical Society_ , Series B:947–1012, 2016.
  * Ahuja et al. [2021] Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish.  Invariance principle meets information bottleneck for out-of-distribution generalization.  _Advances in Neural Information Processing Systems_ , 34:3438–3450, 2021.
  * Arjovsky [2020] Martin Arjovsky.  _Out of distribution generalization in machine learning_.  PhD thesis, New York University, 2020.
  * Arjovsky et al. [2019] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Bareinboim and Pearl [2013] Elias Bareinboim and Judea Pearl.  A general algorithm for deciding transportability of experimental results.  _Journal of causal Inference_ , 1(1):107–134, 2013.
  * Bareinboim and Pearl [2016] Elias Bareinboim and Judea Pearl.  Causal inference and the data-fusion problem.  _Proceedings of the National Academy of Sciences_ , 113(27):7345–7352, 2016.
  * Belkind [2004] Ori Belkind.  Physical systems: conceptual pathways between spacetime and matter.  01 2004.
  * Bengio et al. [2013] Y. Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ , 35:1798–1828, 08 2013.  doi: 10.1109/TPAMI.2013.50.
  * Burgin [2016] Mark Burgin.  _Theory of knowledge: Structures and processes_.  12 2016.  doi: 10.1142/8893.
  * Busch and Lahti [2008] Paul Busch and Pekka Lahti.  Observable (compendium entry), July 2008.  URL http://philsci-archive.pitt.edu/4109/.
  * Coates and Ng [2011] Adam Coates and Andrew Y Ng.  The importance of encoding versus training with sparse coding and vector quantization.  In _Proceedings of the 28th international conference on machine learning (ICML-11)_ , pages 921–928, 2011.
  * Cortes et al. [2019] Corinna Cortes, Mehryar Mohri, and Andrés Muñoz Medina.  Adaptation based on generalized discrepancy.  _Journal of Machine Learning Research_ , 20(1):1–30, 2019.  URL http://jmlr.org/papers/v20/15-192.html.
  * Degtiar and Rose [2023] Irina Degtiar and Sherri Rose.  A review of generalizability and transportability.  _Annual Review of Statistics and its Application_ , 10:501–524, 2023.
  * [14] AP Dempster, NM Laird, and DB Rubin.  Maximal likelihood form incomplete data via the em algorithm.  _J. Roy. Statist. Soc_ , pages 1–38.
  * Dodge et al. [2020] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _arXiv preprint arXiv:2002.06305_ , 2020.
  * Donner [1982] Allan Donner.  The relative effectiveness of procedures commonly used in multiple regression analysis for dealing with missing values.  _The American Statistician_ , 36(4):378–381, 1982\.
  * Evans and Didelez [2021] Robin J Evans and Vanessa Didelez.  Parameterizing and simulating from causal models.  _arXiv preprint arXiv:2109.03694_ , 2021.
  * Ganin et al. [2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030, 2016.
  * Gisin et al. [2002] Nicolas Gisin, Grégoire Ribordy, Wolfgang Tittel, and Hugo Zbinden.  Quantum cryptography.  _Reviews of modern physics_ , 74(1):145, 2002\.
  * Gresele et al. [2022] Luigi Gresele, Julius von Kügelgen, Jonas M. Kübler, Elke Kirschbaum, Bernhard Schölkopf, and Dominik Janzing.  Causal inference through the structural causal marginal problem.  2 2022.  URL http://arxiv.org/abs/2202.01300.
  * Guo et al. [2022] Siyuan Guo, Viktor Tóth, Bernhard Schölkopf, and Ferenc Huszár.  Causal de finetti: On the identification of invariant causal structure in exchangeable data.  _arXiv preprint arXiv:2203.15756_ , 2022.
  * Heinze-Deml et al. [2018] Christina Heinze-Deml, Jonas Peters, and Nicolai Meinshausen.  Invariant causal prediction for nonlinear models.  _Journal of Causal Inference_ , 2018.
  * Hey et al. [2009] Tony Hey, Stewart Tansley, and Kristin Tolle, editors.  _The Fourth Paradigm: Data-Intensive Scientific Discovery_.  Microsoft Research, Redmond, Washington, 2009.  ISBN 978-0-9825442-0-4.  URL https://www.microsoft.com/en-us/research/publication/fourth-paradigm-data-intensive-scientific-discovery/.
  * Higgins et al. [2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  In _International conference on learning representations_ , 2017.
  * Hoffman et al. [2014] Judy Hoffman, Sergio Guadarrama, Eric S Tzeng, Ronghang Hu, Jeff Donahue, Ross Girshick, Trevor Darrell, and Kate Saenko.  Lsda: Large scale detection through adaptation.  _Advances in neural information processing systems_ , 27, 2014.
  * Janzing [2018] Dominik Janzing.  Merging joint distributions via causal model classes with low vc dimension, 2018.  URL https://arxiv.org/abs/1804.03206.
  * Janzing and Schölkopf [2010] Dominik Janzing and Bernhard Schölkopf.  Causal inference using the algorithmic Markov condition.  _IEEE Transactions on Information Theory_ , 56(10):5168–5194, 2010.
  * Jolliffe [2005] Ian Jolliffe.  Principal component analysis.  _Encyclopedia of statistics in behavioral science_ , 2005.
  * Kim and Curry [1977] Jae-On Kim and James Curry.  The treatment of missing data in multivariate analysis.  _Sociological Methods & Research_, 6(2):215–240, 1977.
  * Koller and Friedman [2010] Daphne Koller and Nir Friedman.  _Probabilistic Graphical Models: Principles and Techniques_.  2010\.
  * Krueger et al. [2021] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville.  Out-of-distribution generalization via risk extrapolation (rex).  In _International Conference on Machine Learning_ , pages 5815–5826. PMLR, 2021.
  * Locatello et al. [2019] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.  Challenging common assumptions in the unsupervised learning of disentangled representations.  In _international conference on machine learning_ , pages 4114–4124. PMLR, 2019.
  * Long et al. [2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In _International conference on machine learning_ , pages 97–105. PMLR, 2015.
  * Long et al. [2018] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan.  3 conditional adversarial domain adaptation.  2018\.
  * Lu et al. [2021] Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf.  Invariant causal representation learning for out-of-distribution generalization.  In _International Conference on Learning Representations_ , 2021.
  * Mejia et al. [2021] Sergio Hernan Garrido Mejia, Elke Kirschbaum, and Dominik Janzing.  Obtaining causal information by merging datasets with maxent.  7 2021.  URL http://arxiv.org/abs/2107.07640.
  * Norris [2013] James R. Norris.  Probability and measure.  2013\.
  * Oquab et al. [2014] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic.  Learning and transferring mid-level image representations using convolutional neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1717–1724, 2014.
  * Pearl [2009] Judea Pearl.  _Causality_.  Cambridge university press, 2009.
  * Pearl and Bareinboim [2011] Judea Pearl and Elias Bareinboim.  Transportability across studies: A formal approach.  Technical report, CALIFORNIA UNIV LOS ANGELES DEPT OF COMPUTER SCIENCE, 2011.
  * Pearl and Bareinboim [2022] Judea Pearl and Elias Bareinboim.  External validity: From do-calculus to transportability across populations.  In _Probabilistic and causal inference: The works of Judea Pearl_ , pages 451–482. 2022.
  * Peters et al. [2016] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen.  Causal inference by using invariant prediction: identification and confidence intervals.  _Journal of the Royal Statistical Society. Series B (Statistical Methodology)_ , pages 947–1012, 2016.
  * Pfister et al. [2019] Niklas Pfister, Peter Bühlmann, and Jonas Peters.  Invariant causal prediction for sequential data.  _Journal of the American Statistical Association_ , 114(527):1264–1276, 2019.
  * Robins [1999] James M Robins.  Association, causation, and marginal structural models.  _Synthese_ , 121(1/2):151–179, 1999.
  * Rojas-Carulla et al. [2018] Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters.  Invariant models for causal transfer learning, 2018.  URL http://jmlr.org/papers/v19/16-432.html.
  * Rubin [1976] Donald B Rubin.  Inference and missing data.  _Biometrika_ , 63(3):581–592, 1976.
  * Schölkopf et al. [2012] B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. M. Mooij.  On causal and anticausal learning.  pages 1255–1262, 2012.
  * Schölkopf [2019] Bernhard Schölkopf.  Causality for machine learning.  _arXiv preprint 1911.10500, published: R. Dechter, J. Halpern, and H. Geffner. Probabilistic and Causal Inference: The Works of Judea Pearl. ACM Books_, 2019.
  * Schölkopf et al. [2021] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio.  Towards causal representation learning, 2021.  URL https://arxiv.org/abs/2102.11107.
  * Seneviratne et al. [2018] Martin G. Seneviratne, Michael G. Kahn, and Tina Hernandez-Boussard.  Merging heterogeneous clinical data to enable knowledge discovery.  _Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing_ , 24:439 – 443, 2018.
  * Sermanet et al. [2013] Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun.  Overfeat: Integrated recognition, localization and detection using convolutional networks.  _arXiv preprint arXiv:1312.6229_ , 2013.
  * Shen et al. [2021] Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui.  Towards out-of-distribution generalization: A survey.  _arXiv preprint arXiv:2108.13624_ , 2021.
  * Sklar [1996] Abe Sklar.  Random variables, distribution functions, and copulas: a personal look backward and forward.  _Lecture notes-monograph series_ , pages 1–14, 1996.
  * Spivak [2008] Michael Spivak.  _Calculus_.  Publish or Perish, fourth edition, 2008.
  * Suter et al. [2019] R. Suter, D. Miladinovic, B. Schölkopf, and S. Bauer.  Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness.  In _Proceedings of the 36th International Conference on Machine Learning (ICML)_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 6056–6065. PMLR, June 2019.  URL http://proceedings.mlr.press/v97/suter19a.html.
  * Tzeng et al. [2015] Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko.  Simultaneous deep transfer across domains and tasks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4068–4076, 2015.
  * Vapnik [1999] Vladimir Vapnik.  _The nature of statistical learning theory_.  Springer science & business media, 1999.
  * Vorob’ev [1962] N. N. Vorob’ev.  Consistent families of measures and their extensions.  _Theory of Probability & its Applications_, 7(2):147–163, 1962.  doi: 10.1137/1107014.  URL https://doi.org/10.1137/1107014.
  * Weiss et al. [2016] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang.  A survey of transfer learning.  _Journal of Big data_ , 3(1):1–40, 2016.
  * Wenzel et al. [2022] Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, Bernhard Schölkopf, and Francesco Locatello.  Assaying out-of-distribution generalization in transfer learning.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 7181–7198. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/2f5acc925919209370a3af4eac5cad4a-Paper-Conference.pdf.
  * White [1982] Halbert White.  Maximum likelihood estimation of misspecified models.  _Econometrica_ , 50(1):1–25, 1982.  ISSN 00129682, 14680262.
  * Wildberger et al. [2023] Jonas Wildberger, Siyuan Guo, Arnab Bhattacharyya, and Bernhard Schölkopf.  On the interventional kullback-leibler divergence.  _arXiv preprint arXiv:2302.05380_ , 2023.
  * Yosinski et al. [2014] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  _Advances in neural information processing systems_ , 27, 2014.
  * Zhang et al. [2021a] Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville.  Can subnetwork structure be the key to out-of-distribution generalization?  In _International Conference on Machine Learning_ , pages 12356–12367. PMLR, 2021a.
  * Zhang et al. [2015] K. Zhang, M. Gong, and B. Schölkopf.  Multi-source domain adaptation: A causal view.  In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence_ , pages 3150–3157. AAAI Press, 2015.  URL http://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/10052/9994.
  * Zhang et al. [2021b] Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue He, and Zheyan Shen.  Deep stable learning for out-of-distribution generalization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5372–5382, 2021b.

##  Appendix A Random Variables

In probability measure theory, a random variable is formally defined as below.
Here we inherit the notation and interpretation from [37].

######  Definition 4 (Random Variable)

Let $(\Omega,\mathcal{F},\mathbb{P})$ be a probability space and
$(E,\mathcal{E})$ be a measurable space. A measurable function
$X:\Omega\rightarrow E$ is called a random variable in E.

Intuitively, one can consider the random variable as a function mapping the
outcome of an experiment to some real number, where $E=\mathbb{R}$. The
probability space $(\Omega,\mathcal{F},\mathbb{P})$ is a way to model an
experiment whose outcome is subject to chance, where one can interpret:

$\displaystyle\Omega\ \text{is the set of possible outcomes, or {sample
space}}$ $\displaystyle\mathcal{F}\ \text{is the set of observable sets of
outcomes, or {events} }$ $\displaystyle\mathbb{P}(A)\ \text{is the probability
of an event $A$}$

The law or distribution of $X$ is defined as $\mu_{X}=\mathbb{P}\circ X^{-1}$.
For example, for some real-valued $X$, the distribution function of $X$ is
officially defined as: $F_{X}(x)=\mu_{X}((-\infty,x])=\mathbb{P}(X\leq x)$.

Though well-defined as a mathematical object, it is unclear what does it mean
for two random variables to be the same. It is known that two random variables
can have identical distributions or further same probability space, but
differs significantly. For example, suppose the experiment is to roll a dice
twice. Let $X_{1}$ denotes the outcome of dice in the first roll and $X_{2}$
denotes the outcome of dice in the second roll. Though identically
distributed, $X_{1}$ and $X_{2}$ are independent from each other.

The difference between statistical and physical perspective of random
variables can be seen below. Consider $X_{1}$ and $X_{2}$ are blood pressures
measured in different age groups. It is expected that they will have different
distributions, as elders tend to have higher blood pressures. Mathematically,
$X_{1}$ and $X_{2}$ represent different random variables though they both
represent the same physical property - blood pressures. Under certain
pathological cases, one can also construct two random variables that represent
different physical meanings but have the same mathematical object. For
example, a horse’s body temperature is between $37.5-38.5$ degree Celsius.
Assume a population of horse that has body temperature uniformly distributed
in this range. We know this temperature coincides with humans that have low-
grade fever. Suppose further there is also a population of human patients that
has body temperature uniformly distributed in this range. Random sampling from
the two populations form random variables that share the same probability
space though they represent temperature on different species.

In this work, we borrow well-established terminology from physics to specify
what it means for two random variables are the same. Particularly, we restrict
our attention to random variables that are observable [10], i.e., a physical
property that can be measured. In quantum mechanics, the process that extracts
information about the physical properties of a system is called measurement
process. It is well-known that measurement process could affect the underlying
physical properties of a system [19]. Thus, in this work, we consider two
random variables are the same if they represent the same physical property of
the same physical system and have the same measurement process.

Some examples:

  1. 1.

Suppose in source environment, we observe blood pressured measured from
elderly population. In the target environment, we observe blood pressured
measured from the young. If the blood pressures measured share the same
measurement process, we will consider the differences between environments as
distribution shifts rather than out-of-variable scenarios.

  2. 2.

Suppose a patient’s first day’s blood pressure ($X_{1}$) and oxygen saturation
($X_{2}$) constitutes as important predictors for the length of his/her stay
in ICU ($Y$). If we artificially construct a third variable $X_{3}$ that is a
linear combination of $X_{1}$ and $X_{2}$, e.g. $X_{3}=aX_{1}+bX_{2}$, for
some constant $a,b$. We note, either observing $(X_{1},X_{2})$ or observing
$(X_{2},X_{3})$ allows the same prediction power to the target variable $Y$.
Here we will consider them as out-of-variable scenarios since different
variables can contain the same information.

##  Appendix B Näive Model

Consider the structural causal model described in § 3.1 for ease of
illustration. The method described below applies if we replace variable
$X_{i}$ to any subsets of variables. Consider the true generating function
$\phi$ is additively composed with univariate functions about its covariates,
i.e., there is no interaction term among different covariates. Then the
structural causal model becomes:

$Y:=f_{1}(X_{1})+f_{2}(X_{2})+f_{3}(X_{3})+\epsilon$ (12)

where $X_{1},X_{2},X_{3}$ are jointly independent of each other and
$\epsilon\sim\mathcal{N}(0,\sigma^{2})$. Further, $f_{1},f_{2},f_{3}$ are some
unknown functions.

There are two environments:

  * •

Source env $S$ contains variables $(X_{1},X_{2},Y)$

  * •

Target env $T$ contains variables $(X_{2},X_{3},Y)$

Goal is to learn the most efficient way to transfer knowledge from env $S$ to
env $T$ such that we can learn better predictor $\mathbb{E}[Y\mid
X_{2},X_{3}]$.

By the nature of no-interaction assumption, the solution is easy. One approach
is to build separate neural networks for datasets $(X_{1},Y)$ and $(X_{2},Y)$.
With sufficient data, the learnt functions will be $f_{1},f_{2}$ respectively
due to independence between $X_{i}$. One solution to transfer is to migrate
the learnt function $f_{2}$ to the target environment and then use data from
the target environment to learn $f_{3}$, i.e., functions that involve
previously unobserved variable. Such solution will not hold if there is
interaction between covariates, as function learnt is not an independent
module with respect to the unobserved variable. Here, we highlight the ability
to additive compose different modules relies on the assumption of no-
interaction between unobserved variables and chosen variables.

##  Appendix C Proofs

###  C.1 Impossibility Theorem

See 3.1

###### Proof 1

Proof by construction. Consider the scenario illustrated in Fig. 6(b). Find
two distinct values in the support set of $p(x_{3})$, $x_{3}^{\prime}$ and
$x_{3}^{\prime\prime}$. For some appropriate $\epsilon>0$, consider their
neighbourhoods as $N_{1}=[x_{3}^{\prime}-\epsilon,x_{3}^{\prime}+\epsilon]$
and $N_{2}=[x_{3}^{\prime\prime}-\epsilon,x_{3}^{\prime\prime}+\epsilon]$.
Suppose we learnt the optimal predictive function in the source environment
$f_{A}$, it can be written as:

$\displaystyle f_{A}(x_{1},x_{2})$
$\displaystyle=\int_{\Omega}\phi(x_{1},x_{2},x_{3})p(x_{3})d(x_{3})$ (13)
$\displaystyle=\underbrace{\int_{(\Omega\backslash N_{1})\backslash
N_{2}}\phi(x_{1},x_{2},x_{3})p(x_{3})d(x_{3})}_{\text{Remainder}(x_{1},x_{2})}$
(14)
$\displaystyle+\underbrace{\int_{N_{1}}\phi(x_{1},x_{2},x_{3})p(x_{3})d(x_{3})}_{g(x_{1},x_{2})}+\underbrace{\int_{N_{2}}\phi(x_{1},x_{2},x_{3})p(x_{3})d(x_{3})}_{h(x_{1},x_{2})}$
(15)

We call the integration in the region excluding the specified neighbourhoods
as $\text{Remainder}(x_{1},x_{2})$ and the integration over $N_{1}$ as
$g(x_{1},x_{2})$ and that over $N_{2}$ as $h(x_{1},x_{2})$. Given any function
$c(x_{1},x_{2})$, it is easy to find a function $d(x_{1},x_{2})$ such that
$f_{A}(x_{1},x_{2})-\text{Remainder}(x_{1},x_{2})=c(x_{1},x_{2})g(x_{1},x_{2})+d(x_{1},x_{2})h(x_{1},x_{2})$.
This means whenever we find a function $\phi$ that satisfies Equation (13), it
is always possible to slightly perturb $\phi$ such that $\phi^{\prime}$ can
also satisfy marginal consistency. For example, construct a $\phi^{\prime}$
which is a result of proposed $\phi$ scaled by $c$ elementwise over the
neighbourhood $N_{1}$ and scaled by $d$ elementwise over the neighbourhood
$N_{2}$. Moreover, the deviation of $\phi^{\prime}$ with $\phi$ can be
arbitrarily different:

$\displaystyle||\phi-\phi^{\prime}||_{2}\geq\text{const}_{1}||c-1||_{2}+\text{const}_{2}||d-1||_{2}$
(16)

which the lower bound can be arbitrarily large by the choice of
$c(x_{1},x_{2})$.

Consider the scenario illustrated in Fig. 6(a). Given the information obtained
from the source environment $p(x_{1}),p(x_{2}),f_{A}(x_{1},x_{2})$, by
argument above, we know it is always possible to perturb learnt $\phi$
appropriately to get $\phi^{\prime}$ that satisfies the desired marginal
consistency conditions. We will show it will also be impossible to identify
the optimal predictive function $f_{B}$ in the target environment. By the
argument above, we can choose the function $c(x_{1},x_{2})$ freely. Then there
always exists a function $c$ such that
$\text{sgn}(c(x_{1},x_{2}))=\text{sgn}(\phi(x_{1},x_{2},x_{3}^{\prime}))$ and
$|c(x_{1},x_{2})|\geq L,\forall x_{1},x_{2}$, where $L>1$. Consider a point
$x_{3}^{\prime}$ in the neighbourhood $N_{1}$. Then under a learnt $\phi$ and
perturbed $\phi^{\prime}$, its corresponding optimal predictive function
$f_{B}$ and $f_{B}^{\prime}$ can be written as:

$\displaystyle f_{B}(x_{2},x_{3}^{\prime})$
$\displaystyle=\int_{\Omega}\phi(x_{1},x_{2},x_{3}^{\prime})p(x_{1})d(x_{1})$
(17) $\displaystyle f_{B}^{\prime}(x_{2},x_{3}^{\prime})$
$\displaystyle=\int_{\Omega}\underbrace{c(x_{1},x_{2})\phi(x_{1},x_{2},x_{3}^{\prime})}_{\geq
0\ \text{and}\ \neq\phi}p(x_{1})d(x_{1})$ (18)

This implies $f_{B}(x_{2},x_{3}^{\prime})\neq
f_{B}^{\prime}(x_{2},x_{3}^{\prime})$ for all values in the neighbourhood
$N_{1}$. This means though $f_{B}$ and $f_{B}^{\prime}$ are both marginally
consistent with $f_{A}$ (since $\phi$ and $\phi^{\prime}$ are both consistent
with $f_{A}$), but they are different functions. Moreover their difference can
be arbitrarily large:

$\displaystyle||f_{B}-f_{B}^{\prime}||_{2}\geq\int\int_{N_{1}}(f_{B}(x_{2},x_{3})-f_{B}^{\prime}(x_{2},x_{3}))^{2}p(x_{3})dx_{3}p(x_{2})dx_{2}$
(19)

If analyse the term that squared, we have:

$\displaystyle(f_{B}(x_{2},x_{3})-f_{B}^{\prime}(x_{2},x_{3}))^{2}$
$\displaystyle=(\int_{\Omega}(c-1)\phi(x_{1},x_{2},x_{3})p(x_{1})dx_{1})^{2}\geq(|L|-1)^{2}(\int_{\Omega}\phi
p(x_{1})dx_{1})^{2}$

The last inequality holds by construction of $c$. Thus substituting it into
Eq. 19, we have $||f_{B}-f_{B}^{\prime}||_{2}\geq\text{const}*(|L|-1)^{2}$,
where the lower bound of the constructed function $c$ can be arbitrarily
large.

Consider the scenario illustrated in Fig. 6(c). Here we restrict to cases when
$X_{1}$ and $X_{3}$ are binary variables. Set $\gamma_{i}:=P(X_{i}=0)$. Then
given source environments where one observes variables $(X_{1},X_{2},Y)$ and
the other observes variables $(X_{2},X_{3},Y)$. The potential generating
function must satisfy below system of equations:

$\displaystyle f_{A}(0,x_{2})$
$\displaystyle=\gamma_{3}\phi(0,x_{2},0)+(1-\gamma_{3})\phi(0,x_{2},1)$ (20)
$\displaystyle f_{A}(1,x_{2})$
$\displaystyle=\gamma_{3}\phi(1,x_{2},0)+(1-\gamma_{3})\phi(1,x_{2},1)$ (21)
$\displaystyle f_{B}(x_{2},0)$
$\displaystyle=\gamma_{1}\phi(0,x_{2},0)+(1-\gamma_{1})\phi(1,x_{2},0)$ (22)
$\displaystyle f_{B}(x_{2},1)$
$\displaystyle=\gamma_{1}\phi(0,x_{2},1)+(1-\gamma_{1})\phi(1,x_{2},1)$ (23)

Perturb $\phi(0,x_{2},0)$ by $c(0,x_{2},0)$, then in order to still satisfy
the above system of equations, the coefficients need to be correspondingly
adjusted as:

$\displaystyle c(0,x_{2},1)$
$\displaystyle=\frac{f_{A}(0,x_{2})-\gamma_{3}c(0,x_{2},0)\phi(0,x_{2},0)}{(1-\gamma_{3})\phi(0,x_{2},1)}$
(24) $\displaystyle c(1,x_{2},1)$
$\displaystyle=\frac{f_{B}(x_{2},1)-\frac{\gamma_{1}f_{A}(0,x_{2})-\gamma_{1}\gamma_{3}c(0,x_{2},0)\phi(0,x_{2},0)}{(1-\gamma_{3})}}{(1-\gamma_{1})\phi(1,x_{2},1)}$
(25) $\displaystyle c(1,x_{2},0)$
$\displaystyle=\frac{f_{A}(1,x_{2})-\frac{(1-\gamma_{3})f_{B}(x_{2},1)-\gamma_{1}f_{A}(0,x_{2})+\gamma_{1}\gamma_{3}c(0,x_{2},0)\phi(0,x_{2},0)}{1-\gamma_{1}}}{\gamma_{3}\phi(1,x_{2},0)}$
(26)

Note we have the adjusted coefficients are consistent with each other:

$\displaystyle(1-\gamma_{1})c(1,x_{2},0)\phi(1,x_{2},0)$
$\displaystyle=\frac{1}{\gamma_{3}}\big{[}(1-\gamma_{1})f_{A}(1,x_{2})-(1-\gamma_{3})f_{B}(x_{2},1)$
(27)
$\displaystyle+\gamma_{1}f_{A}(0,x_{2})-\gamma_{1}\gamma_{3}c(0,x_{2},0)\phi(0,x_{2},0)\big{]}$
(28) $\displaystyle=f_{B}(x_{2},0)-\gamma_{1}c(0,x_{2},0)\phi(0,x_{2},0)$ (29)

Thus it is possible to find a new $\phi^{\prime}$ such that it still satisfies
the system of equations. More over $\phi^{\prime}$ deviates from $\phi$
arbitrarily large by the choice of $c(0,x_{2},0)$:

$\displaystyle||\phi-\phi^{\prime}||_{2}\geq||\phi(0,x_{2},0)-c(0,x_{2},0)\phi(0,x_{2},0)||_{2}\geq||c-1||_{2}*\text{const}$
(30)

###  C.2 Proof of possibility

See 3.2

$X_{1}$$X_{2}$$Y$

Figure 7: An example of scenarios considered in Theorem 3.2

###### Proof 2

The scenario illustrated satisfy

$\displaystyle Y:=\phi(\textbf{PA}_{Y})+\epsilon_{Y}$ (31) $\displaystyle
Z:=g(\textbf{PA}_{Y})+\epsilon_{1}$ (32) $\displaystyle X_{2}:=U_{2}$ (33)

where $\phi$ and $g$ are some reasonably continuous functions and $g$ is
invertible, $\epsilon_{Y}\sim\mathcal{N}(0,\sigma_{Y}^{2})$ and
$\epsilon_{1}\sim\mathcal{N}(0,\sigma_{1}^{2})$. The SCM is illustrated by the
DAG in Fig. 7.

The optimal predictive function in the target environment coincides with the
true generating function, written as:
$f_{B}(\textbf{PA}_{Y})=\mathbb{E}[Y|\textbf{PA}_{Y}]=\phi(\textbf{PA}_{Y})$.
Given $g$ is continuous and invertible, we have $g^{-1}$ exists and is
continuous. Since $\textbf{pa}_{Y}=g^{-1}(z-\epsilon_{1})$, given
$\epsilon_{1}$ is small enough, by continuity in $g^{-1}$, approximate
$\textbf{pa}_{Y}$ as $g^{-1}(z)$. By $Z\perp\\!\\!\\!\perp
Y\mid\textbf{PA}_{Y}$, we have

$\displaystyle p(y\mid z)$ $\displaystyle=\int
p(y\mid\textbf{pa}_{Y},z)p(\textbf{pa}_{Y}\mid z)d\textbf{pa}_{Y}$ (34)
$\displaystyle=\int p(y\mid\textbf{pa}_{Y})p(\textbf{pa}_{Y}\mid
z)d\textbf{pa}_{Y}$ (35)

Performing point estimation over the integral where we assume
$p(\textbf{pa}_{Y}\mid z)=\delta(\textbf{pa}_{Y}=g^{-1}(z))$, we have $p(y\mid
z)=\mathcal{N}(y\mid\phi\cdot g^{-1}(z),\sigma_{Y}^{2})$ and
$f_{S_{2}}(z)=\mathbb{E}[Y\mid z]=\phi\cdot g^{-1}(z)$. This implies, we can
recover the optimal predictive function in the target environment by composing
learnt functions from the two source environments, where
$f_{B}=\phi=f_{S_{1}}\cdot f_{S_{2}}$.

###  C.3 Proof of Theorem 3.1

See 3.1

######  Theorem C.1 (Taylor’s Theorem [54])

Let $k\geq 1$ be an integer and let the function
$f:\mathbb{R}\rightarrow\mathbb{R}$ be $k+1$-times differentiable everywhere
and $f^{(k)}$ is continuous on the closed interval $[a,x]$. Then there exists
a point $\xi\in[a,x]$ such that

$f(x)=f(a)+f^{\prime}(a)(x-a)+\frac{f^{\prime\prime}(a)}{2!}(x-a)^{2}+\dots+\frac{f^{(k)}(a)}{k!}(x-a)^{k}+\underbrace{\frac{f^{(k+1)}(\xi)}{(k+1)!}(x-a)^{k+1}}_{:=R_{k+1}}$
(36)

###### Proof 3

Let $\mu_{x}=\mathbb{E}[X],\sigma_{x}^{2}=\text{Var}[X],k_{x}=\text{Skew}[X]$
and $a=\mu_{x}$, then Theorem C.1 states that

$f(x)=f(\mu_{x})+f^{\prime}(\mu_{x})(x-\mu_{x})+C(x-\mu_{x})^{2}$ (37)

Take first-order Taylor approximation over the generating function, we suppose
$f(x)\approx f(\mu_{x})+f^{\prime}(\mu_{x})(x-\mu_{x})$. Taking expectations,
then

$\mathbb{E}[f(x)]\approx f(\mu_{x})$ (38)

Consider the residuals of $f(x)$ without its expectations and raise it to the
power of $n$, we have:

$\displaystyle\bigg{(}f(x)-\mathbb{E}[f(x)]\bigg{)}^{n}=\bigg{(}$
$\displaystyle f^{\prime}(\mu_{x})(x-\mu_{x})\bigg{)}^{n}$ (39)

Taking expectations, on Eq. 39, we have:

$\displaystyle\mathbb{E}\bigg{[}\big{(}f(x)-\mathbb{E}[f(x)]\big{)}^{n}\bigg{]}=\big{(}f^{\prime}(\mu_{x})\big{)}^{n}\mathbb{E}\big{[}(x-\mu_{x})^{n}\big{]}$
(40)

Take $f_{x_{1},x_{2}}:\mathcal{X}_{3}\rightarrow\mathbb{R}$ to be the function
$\phi:x_{1}\times x_{2}\times\mathcal{X}_{3}\rightarrow\mathbb{R}$ where
values $x_{1},x_{2}$ are fixed. We can substitute $f$ as $f_{x_{1},x_{2}}$ to
Eq. 40. Since $Y=\phi(x_{1},x_{2},x_{3})+\epsilon$, we have

$\displaystyle\mathbb{E}[(Y-f_{S}(x_{1},x_{2}))^{n}\mid x_{1},x_{2}]$
$\displaystyle=\mathbb{E}\bigg{[}\big{(}f(x)+\epsilon-\mathbb{E}[f(x)]\big{)}^{n}\bigg{]}$
(41) $\displaystyle=\sum_{k=0}^{n}{n\choose
k}\mathbb{E}[\epsilon^{k}]\mathbb{E}[(f(x)-\mathbb{E}[f(x)])^{n-k}]$ (42)

The second equation is due to independence of $\epsilon$ and
$f(x)-\mathbb{E}[f(x)]$. This is well known that
$\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]$ if $X$ and $Y$ are independent
from each other.

##  Appendix D Extensions

###  D.1 Higher-order Taylor Approximations

Appendix C.3 shows the proof of Theorem 3.1 where we approximated the function
$\phi:x_{1}\times x_{2}\times\mathcal{X}_{3}\rightarrow\mathbb{R}$ with first-
order Taylor approximations. Here we demonstrate its flexibility to extend to
higher-order Taylor approximations, e.g. second-order and for ease of
illustration and computation, we focus our work on first-order Taylor
approximations. Let
$\mu_{x}=\mathbb{E}[X],\sigma_{x}^{2}=\text{Var}[X],k_{x}=\text{Skew}[X]$ and
$a=\mu_{x}$. Theorem C.1 states that with second order approximations, the
function can be written as:

$\displaystyle
f(x)=f(\mu_{x})+f^{\prime}(\mu_{x})(x-\mu_{x})+f^{\prime\prime}(\mu_{x})(x-\mu_{x})^{2}+C(x-\mu_{x})^{3}$
(43)

The moment of the residuals of $f(x)$ is expressed as:

$\displaystyle\bigg{(}f(x)-\mathbb{E}[f(x)]\bigg{)}^{n}=\bigg{(}$
$\displaystyle
f^{\prime}(\mu_{x})(x-\mu_{x})+f^{\prime\prime}(\mu_{x})\big{(}(x-\mu_{x})^{2}-\sigma_{x}^{2}\big{)}\bigg{)}^{n}$
(44)

When $n=2$ and $n=3$:

$\displaystyle\bigg{(}f(x)-\mathbb{E}[f(x)]\bigg{)}^{2}$
$\displaystyle=f^{\prime}(\mu_{x})^{2}(x-\mu_{x})^{2}+2f^{\prime}(\mu_{x})f^{\prime\prime}(\mu_{x})(x-\mu_{x})\big{(}(x-\mu_{x})^{2}-\sigma_{x}^{2}\big{)}$
$\displaystyle+f^{\prime\prime}(\mu_{x})^{2}\big{(}(x-\mu_{x})^{2}-\sigma_{x}^{2}\big{)}^{2}$
$\displaystyle\bigg{(}f(x)-\mathbb{E}[f(x)]\bigg{)}^{3}$
$\displaystyle=f^{\prime}(\mu_{x})^{3}(x-\mu_{x})^{3}+3f^{\prime}(\mu_{x})^{2}f^{\prime\prime}(\mu_{x})(x-\mu_{x})^{2}\big{(}(x-\mu_{x})^{2}-\sigma_{x}^{2}\big{)}$
$\displaystyle+3f^{\prime}(\mu_{x})f^{\prime\prime}(\mu_{x})^{2}(x-\mu_{x})\big{(}(x-\mu_{x})^{2}-\sigma_{x}^{2}\big{)}^{2}+f^{\prime\prime}(\mu_{x})^{3}\big{(}(x-\mu_{x})^{2}-\sigma_{x}^{2}\big{)}^{3}$

Taking expectations on both sides:

$\displaystyle\mathbb{E}[\big{(}f(x)-\mathbb{E}[f(x)]\big{)}^{2}]$
$\displaystyle=f^{\prime}(\mu_{x})^{2}\sigma_{x}^{2}+2f^{\prime}(\mu_{x})f^{\prime\prime}(\mu_{x})k_{x}+f^{\prime\prime}(\mu_{x})^{2}\big{(}\mathbb{E}[(x-\mu_{x})^{4}]-\sigma_{x}^{4}\big{)}$
(45) $\displaystyle\mathbb{E}[\big{(}f(x)-\mathbb{E}[f(x)]\big{)}^{3}]$
$\displaystyle=f^{\prime}(\mu_{x})^{3}k_{x}+3f^{\prime}(\mu_{x})^{2}f^{\prime\prime}(\mu_{x})\big{(}\mathbb{E}[(x-\mu_{x})^{4}]-\sigma_{x}^{4}\big{)}$
(46)
$\displaystyle+3f^{\prime}(\mu_{x})f^{\prime\prime}(\mu_{x})^{2}\big{(}\mathbb{E}[(x-\mu_{x})^{5}]-2\sigma_{x}^{2}k_{x}\big{)}$
(47)
$\displaystyle+f^{\prime\prime}(\mu_{x})^{3}(\mathbb{E}[(x-\mu_{x})^{6}]-3\sigma_{x}^{2}\mathbb{E}[(x-\mu_{x})^{4}]+2\sigma_{x}^{6})$
(48)

Take $f_{x_{1},x_{2}}:\mathcal{X}_{3}\rightarrow\mathbb{R}$ to be the function
$\phi:x_{1}\times x_{2}\times\mathcal{X}_{3}\rightarrow\mathbb{R}$ where
values $x_{1},x_{2}$ are fixed. We can substitute $f$ as $f_{x_{1},x_{2}}$.
Recall when $n=2$ and $n=3$ in Theorem 3.1,

$\displaystyle\mathbb{E}[(Y-f_{S}(x_{1},x_{2}))^{2}\mid x_{1},x_{2}]$
$\displaystyle=\mathbb{E}[(\phi(x_{1},x_{2},X_{3})-f_{S}(x_{1},x_{2}))^{2}\mid
x_{1},x_{2}]+\mathbb{E}[\epsilon^{2}]$ (49)
$\displaystyle\mathbb{E}[(Y-f_{S}(x_{1},x_{2}))^{3}\mid x_{1},x_{2}]$
$\displaystyle=\mathbb{E}[(\phi(x_{1},x_{2},X_{3})-f_{S}(x_{1},x_{2}))^{3}\mid
x_{1},x_{2}]+\mathbb{E}[\epsilon^{3}]$ (50)

Assume $\epsilon$ has small variance and negligible skew and given we can
estimate the higher moments of variable $X_{3}$, with two equations and two
unknowns, we are able to estimate the unknowns, namely
$\frac{\partial\phi}{\partial X_{3}}\big{|}_{x_{1},x_{2},\mu_{3}}$ and
$\frac{\partial^{2}\phi}{\partial^{2}X_{3}}\big{|}_{x_{1},x_{2},\mu_{3}}$.

###  D.2 More than one unobserved variables

Here, we consider a function with two variables $f(x,y)$ where variables can
be considered as two unobserved variables from the source environment. Note,
the same argument can easily extend to multivariate functions. Let
$\mathbb{E}[X]=\mu_{x},\mathbb{E}[Y]=\mu_{y}$. Expand multivariate Taylor
approximations around the point $\mathbf{a}=(\mu_{x},\mu_{y})$, we have:

$\displaystyle f(x,y)=$ $\displaystyle f(\mu_{x},\mu_{y})+\frac{\partial
f}{\partial x}\big{|}_{\mathbf{a}}(x-\mu_{x})+\frac{\partial f}{\partial
y}\big{|}_{\mathbf{a}}(y-\mu_{y})$ (51)
$\displaystyle+C_{1}(x-\mu_{x})^{2}+C_{2}(x-\mu_{x})(y-\mu_{y})+C_{3}(y-\mu_{y})^{3}$
(52)

With first-order Taylor approximations, we ignore the higher order terms.
Taking expectations on both sides, we have
$\mathbb{E}[f(x,y)]=f(\mu_{x},\mu_{y})$. Similarly,

$\displaystyle\big{(}f(x,y)-\mathbb{E}[f(x,y)]\big{)}^{n}$
$\displaystyle=\big{(}\frac{\partial f}{\partial
x}\big{|}_{\mathbf{a}}(x-\mu_{x})+\frac{\partial f}{\partial
y}\big{|}_{\mathbf{a}}(y-\mu_{y})\big{)}^{n}$ (53)
$\displaystyle=\sum_{k=0}^{n}{n\choose k}\big{(}\frac{\partial f}{\partial
x}\big{|}_{\mathbf{a}}\big{)}^{k}(x-\mu_{x})^{k}\big{(}\frac{\partial
f}{\partial y}\big{|}_{\mathbf{a}}\big{)}^{n-k}(y-\mu_{y})^{n-k}$ (54)

Taking expectations on both sides, assuming we can estimate the cross-moments
between two unobserved variables from data, with two unknowns and two
equations, we can estimate the unknowns. Note here we also relax the
independence assumption among the unobserved variables.

##  Appendix E Algorithm

To estimate the target predictive function, one can consider it as the true
generating function marginalized over the unobserved variable $X_{1}$ in
environment $T$. The optimal predictive function in the target environment can
be evaluated as:

$\displaystyle f_{T}(x_{2},x_{3})$
$\displaystyle=\int\phi(x_{1},x_{2},x_{3})p(x_{1})dx_{1}\approx\frac{1}{n}\sum_{i=1}^{n}\phi(x_{1,i},x_{2},x_{3}),\quad\text{where}\
x_{1,i}\sim p(x_{1})$

If the true-generating function is known, the above term can be Monte Carlo
estimated as environment $S$ contains the marginal distribution of
$X_{1}$.Though identifying the generating function is hard, Theorem 3.1 shows
that we can approximate it with first-degree Taylor approximation around its
third variable:

$\displaystyle\phi(x_{1},x_{2},x_{3})$
$\displaystyle=\phi(x_{1},x_{2},\mu_{3})+\frac{\partial\phi}{\partial
X_{3}}\bigg{|}_{x_{1},x_{2},\mu_{3}}(x_{3}-\mu_{3})+C(x_{3}-\mu_{3})^{2}$ (55)

for some constant $C$. Taking expectations over $X_{3}$ on both sides, we have
$f_{S}(x_{1},x_{2})=\phi(x_{1},x_{2},\mu_{3})+C\sigma_{3}^{2}$. Approximating
the first term in Eq. 55 with $f_{S}(x_{1},x_{2})$ and estimate the partial
derivative term by Theorem 3.1 gives us Algorithm 1.

##  Appendix F Data Generation Details

We generate synthetic data according to the following equations:

$\displaystyle\forall
i\in[3]:X_{i}\overset{i.i.d}{\sim}\Gamma(1,1),\tilde{X}_{i}\leftarrow\text{standardize}(X_{i})$
(56) $\displaystyle\forall
j\in[7]:\alpha_{j}\overset{i.i.d}{\sim}\mathcal{N}(0,1)$ (57) $\displaystyle
Y_{poly}=\alpha^{T}g_{poly}(\tilde{X}_{1},\tilde{X}_{2},\tilde{X}_{3})+\epsilon_{1}$
(58) $\displaystyle
Y_{nonlinear}=\sqrt{(\alpha_{1}\tilde{X}_{1})^{2}+(\alpha_{2}\tilde{X}_{2})^{2}+(\alpha_{3}\tilde{X}_{3})^{2}}+\epsilon_{2}$
(59) $\displaystyle
Y_{trig}=\alpha_{1}\cos(\tilde{X}_{1})+\alpha_{2}\sin(\tilde{X}_{2})+\alpha_{3}\cos(\tilde{X}_{3})+\alpha_{4}\cos(\tilde{X}_{1}\tilde{X}_{3})+\epsilon_{3}$
(60)

where
$g_{poly}(X_{1},X_{2},X_{3})=[X_{1},X_{2},X_{3},X_{1}X_{2},X_{1}X_{3},X_{2}X_{3}.X_{1}X_{2}X_{3}]$
and $\epsilon_{i}\sim\mathcal{N}(0,\sigma^{2})$ for all $i$ and $\sigma=0.1$.
The covariates are standardized into range between $0$ and $5$.